2023-01-11T21:14:24.9309637Z Requested labels: linux.8xlarge.nvidia.gpu
2023-01-11T21:14:24.9309732Z Job defined at: pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/trunk/91627
2023-01-11T21:14:24.9309867Z Reusable workflow chain:
2023-01-11T21:14:24.9309904Z pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/91627 (8419ddda87c8a47eacc63b54bc7ec98c1f27c26e)
2023-01-11T21:14:24.9309947Z -> pytorch/pytorch/.github/workflows/_linux-test.yml@refs/tags/ciflow/trunk/91627 (8419ddda87c8a47eacc63b54bc7ec98c1f27c26e)
2023-01-11T21:14:24.9310000Z Waiting for a runner to pick up this job...
2023-01-11T21:14:25.2467756Z Job is about to start running on the runner: i-0f914c3983ac93cd3 (organization)
2023-01-11T21:14:30.3443000Z Current runner version: '2.300.2'
2023-01-11T21:14:30.3451287Z Runner name: 'i-0f914c3983ac93cd3'
2023-01-11T21:14:30.3451828Z Runner group name: 'Default'
2023-01-11T21:14:30.3452511Z Machine name: 'ip-10-0-4-67'
2023-01-11T21:14:30.3454771Z ##[group]GITHUB_TOKEN Permissions
2023-01-11T21:14:30.3455533Z Actions: write
2023-01-11T21:14:30.3455919Z Checks: write
2023-01-11T21:14:30.3456280Z Contents: write
2023-01-11T21:14:30.3456648Z Deployments: write
2023-01-11T21:14:30.3456968Z Discussions: write
2023-01-11T21:14:30.3457342Z Issues: write
2023-01-11T21:14:30.3457730Z Metadata: read
2023-01-11T21:14:30.3458039Z Packages: write
2023-01-11T21:14:30.3458403Z Pages: write
2023-01-11T21:14:30.3458805Z PullRequests: write
2023-01-11T21:14:30.3459165Z RepositoryProjects: write
2023-01-11T21:14:30.3459600Z SecurityEvents: write
2023-01-11T21:14:30.3459972Z Statuses: write
2023-01-11T21:14:30.3460297Z ##[endgroup]
2023-01-11T21:14:30.3464537Z Secret source: Actions
2023-01-11T21:14:30.3465418Z Prepare workflow directory
2023-01-11T21:14:30.6890600Z Prepare all required actions
2023-01-11T21:14:30.7124836Z Getting action download info
2023-01-11T21:14:31.3568722Z Download action repository 'pytorch/test-infra@main' (SHA:2c225610d00fb13c04fcd60389d3e4d8326167c3)
2023-01-11T21:14:31.6786351Z Download action repository 'pytorch/pytorch@master' (SHA:c5836153f5332ca83d5cacde38f2829a4d54793e)
2023-01-11T21:14:35.0924979Z Download action repository 'seemethere/upload-artifact-s3@v5' (SHA:baba72d0712b404f646cebe0730933554ebce96a)
2023-01-11T21:14:35.4073768Z Getting action download info
2023-01-11T21:14:35.9744134Z Download action repository 'malfet/checkout@silent-checkout' (SHA:c7b8fef48edfe1bca0044a44b1f7f7c4318a3076)
2023-01-11T21:14:36.1621240Z Getting action download info
2023-01-11T21:14:36.6890818Z Download action repository 'nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482' (SHA:3e91a01664abd3c5cd539100d10d33b9c5b68482)
2023-01-11T21:14:36.8402164Z Uses: pytorch/pytorch/.github/workflows/_linux-test.yml
2023-01-11T21:14:36.8404976Z ##[group] Inputs
2023-01-11T21:14:36.8405384Z   build-environment: linux-bionic-cuda11.7-py3.10-gcc7
2023-01-11T21:14:36.8406834Z   test-matrix: { include: [
  { config: "default", shard: 1, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 2, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 3, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "default", shard: 4, num_shards: 4, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "functorch", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "nogpu_AVX512", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
  { config: "nogpu_NO_AVX2", shard: 1, num_shards: 1, runner: "linux.2xlarge" },
  { config: "jit_legacy", shard: 1, num_shards: 1, runner: "linux.4xlarge.nvidia.gpu" },
  { config: "distributed", shard: 1, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 2, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
  { config: "distributed", shard: 3, num_shards: 3, runner: "linux.8xlarge.nvidia.gpu" },
]}

2023-01-11T21:14:36.8408354Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:14:36.8408863Z   sync-tag: 
2023-01-11T21:14:36.8409939Z   timeout-minutes: 240
2023-01-11T21:14:36.8410227Z   use-gha: 
2023-01-11T21:14:36.8410500Z ##[endgroup]
2023-01-11T21:14:36.8411327Z Complete job name: linux-bionic-cuda11.7-py3.10-gcc7 / test (distributed, 1, 3, linux.8xlarge.nvidia.gpu)
2023-01-11T21:14:36.9526432Z ##[group]Run pytorch/test-infra/.github/actions/setup-ssh@main
2023-01-11T21:14:36.9526822Z with:
2023-01-11T21:14:36.9527370Z   github-secret: ***
2023-01-11T21:14:36.9527832Z   instructions: All testing is done inside the container, to start an interactive session run:
  docker exec -it $(docker container ps --format '{{.ID}}') bash

2023-01-11T21:14:36.9528287Z   activate-with-label: false
2023-01-11T21:14:36.9528715Z   label: with-ssh
2023-01-11T21:14:36.9528982Z   remove-existing-keys: true
2023-01-11T21:14:36.9529228Z env:
2023-01-11T21:14:36.9529472Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:14:36.9529734Z ##[endgroup]
2023-01-11T21:14:37.0613087Z ciflow reference detected, attempting to extract PR number
2023-01-11T21:14:37.6363134Z Grabbing public ssh keys from https://github.com/pytorch-bot[bot].keys
2023-01-11T21:14:37.7298810Z No SSH keys found for user pytorch-bot[bot]
2023-01-11T21:14:37.7299443Z Grabbing public ssh keys from https://github.com/LucaLumetti.keys
2023-01-11T21:14:37.8777882Z ~/.ssh/authorized_keys file found on node, removing ~/.ssh and starting fresh
2023-01-11T21:14:37.8799126Z Public keys pulled and installed to /home/ec2-user/.ssh/authorized_keys
2023-01-11T21:14:37.8845175Z Login using: ssh ec2-user@ec2-107-23-101-53.compute-1.amazonaws.com
2023-01-11T21:14:37.8846465Z All testing is done inside the container, to start an interactive session run:
2023-01-11T21:14:37.8847172Z   docker exec -it $(docker container ps --format '{{.ID}}') bash
2023-01-11T21:14:37.9129497Z ##[group]Run pytorch/pytorch/.github/actions/checkout-pytorch@master
2023-01-11T21:14:37.9129937Z with:
2023-01-11T21:14:37.9130164Z   submodules: recursive
2023-01-11T21:14:37.9130425Z   fetch-depth: 0
2023-01-11T21:14:37.9130654Z env:
2023-01-11T21:14:37.9130875Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:14:37.9131137Z ##[endgroup]
2023-01-11T21:14:37.9417617Z ##[group]Run retry () {
2023-01-11T21:14:37.9417943Z retry () {
2023-01-11T21:14:37.9418257Z   $* || (sleep 1 && $*) || (sleep 2 && $*) || (sleep 4 && $*) || (sleep 8 && $*)
2023-01-11T21:14:37.9418534Z }
2023-01-11T21:14:37.9418793Z echo "${GITHUB_WORKSPACE}"
2023-01-11T21:14:37.9419099Z if [ -z "${NO_SUDO}" ]; then
2023-01-11T21:14:37.9419397Z   retry sudo rm -rf "${GITHUB_WORKSPACE}"
2023-01-11T21:14:37.9419682Z else
2023-01-11T21:14:37.9419957Z   retry rm -rf "${GITHUB_WORKSPACE}"
2023-01-11T21:14:37.9420220Z fi
2023-01-11T21:14:37.9420513Z mkdir "${GITHUB_WORKSPACE}"
2023-01-11T21:14:37.9439211Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:14:37.9439523Z env:
2023-01-11T21:14:37.9439780Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:14:37.9440037Z   NO_SUDO: 
2023-01-11T21:14:37.9440258Z ##[endgroup]
2023-01-11T21:14:37.9567962Z /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T21:14:41.0142551Z ##[group]Run malfet/checkout@silent-checkout
2023-01-11T21:14:41.0142906Z with:
2023-01-11T21:14:41.0143188Z   ref: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:14:41.0143463Z   fetch-depth: 0
2023-01-11T21:14:41.0143719Z   submodules: recursive
2023-01-11T21:14:41.0143983Z   quiet-checkout: true
2023-01-11T21:14:41.0144242Z   repository: pytorch/pytorch
2023-01-11T21:14:41.0144643Z   token: ***
2023-01-11T21:14:41.0144886Z   ssh-strict: true
2023-01-11T21:14:41.0145157Z   persist-credentials: true
2023-01-11T21:14:41.0145420Z   clean: true
2023-01-11T21:14:41.0145662Z   lfs: false
2023-01-11T21:14:41.0145914Z   set-safe-directory: true
2023-01-11T21:14:41.0146147Z env:
2023-01-11T21:14:41.0146387Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:14:41.0146644Z ##[endgroup]
2023-01-11T21:14:41.1682224Z Syncing repository: pytorch/pytorch
2023-01-11T21:14:41.1684055Z ##[group]Getting Git version info
2023-01-11T21:14:41.1684956Z Working directory is '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2023-01-11T21:14:41.1685573Z [command]/usr/bin/git version
2023-01-11T21:14:41.1685829Z git version 2.38.1
2023-01-11T21:14:41.1704927Z ##[endgroup]
2023-01-11T21:14:41.1725599Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/2e1924ae-aac8-4139-9476-88a594cbbcd6' before making global git config changes
2023-01-11T21:14:41.1727013Z Adding repository directory to the temporary git global config as a safe directory
2023-01-11T21:14:41.1733424Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T21:14:41.1780223Z Deleting the contents of '/home/ec2-user/actions-runner/_work/pytorch/pytorch'
2023-01-11T21:14:41.1787026Z ##[group]Initializing the repository
2023-01-11T21:14:41.1790965Z [command]/usr/bin/git init /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T21:14:41.1825343Z hint: Using 'master' as the name for the initial branch. This default branch name
2023-01-11T21:14:41.1825794Z hint: is subject to change. To configure the initial branch name to use in all
2023-01-11T21:14:41.1826232Z hint: of your new repositories, which will suppress this warning, call:
2023-01-11T21:14:41.1826554Z hint: 
2023-01-11T21:14:41.1826930Z hint: 	git config --global init.defaultBranch <name>
2023-01-11T21:14:41.1827205Z hint: 
2023-01-11T21:14:41.1827594Z hint: Names commonly chosen instead of 'master' are 'main', 'trunk' and
2023-01-11T21:14:41.1828107Z hint: 'development'. The just-created branch can be renamed via this command:
2023-01-11T21:14:41.1828424Z hint: 
2023-01-11T21:14:41.1828853Z hint: 	git branch -m <name>
2023-01-11T21:14:41.1829601Z Initialized empty Git repository in /home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/
2023-01-11T21:14:41.1840678Z [command]/usr/bin/git remote add origin https://github.com/pytorch/pytorch
2023-01-11T21:14:41.1875298Z ##[endgroup]
2023-01-11T21:14:41.1875791Z ##[group]Disabling automatic garbage collection
2023-01-11T21:14:41.1880019Z [command]/usr/bin/git config --local gc.auto 0
2023-01-11T21:14:41.1912310Z ##[endgroup]
2023-01-11T21:14:41.1913289Z ##[group]Setting up auth
2023-01-11T21:14:41.1922788Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2023-01-11T21:14:41.1958350Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2023-01-11T21:14:41.2265031Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2023-01-11T21:14:41.2299016Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2023-01-11T21:14:41.2603487Z [command]/usr/bin/git config --local http.https://github.com/.extraheader AUTHORIZATION: basic ***
2023-01-11T21:14:41.2651547Z ##[endgroup]
2023-01-11T21:14:41.2652044Z ##[group]Fetching the repository
2023-01-11T21:14:41.2660832Z [command]/usr/bin/git -c protocol.version=2 fetch --prune --quiet --no-recurse-submodules origin +refs/heads/*:refs/remotes/origin/* +refs/tags/*:refs/tags/*
2023-01-11T21:15:38.6878035Z [command]/usr/bin/git rev-parse --verify --quiet 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e^{object}
2023-01-11T21:15:38.6908263Z 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:15:38.6916212Z ##[endgroup]
2023-01-11T21:15:38.6916750Z ##[group]Determining the checkout info
2023-01-11T21:15:38.6917231Z ##[endgroup]
2023-01-11T21:15:38.6917681Z ##[group]Checking out the ref
2023-01-11T21:15:38.6921855Z [command]/usr/bin/git checkout --quiet --force 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:15:40.4976826Z ##[endgroup]
2023-01-11T21:15:40.4977558Z ##[group]Setting up auth for fetching submodules
2023-01-11T21:15:40.4983436Z [command]/usr/bin/git config --global http.https://github.com/.extraheader AUTHORIZATION: basic ***
2023-01-11T21:15:40.5040191Z [command]/usr/bin/git config --global --unset-all url.https://github.com/.insteadOf
2023-01-11T21:15:40.5074488Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf git@github.com:
2023-01-11T21:15:40.5109248Z [command]/usr/bin/git config --global --add url.https://github.com/.insteadOf org-21003710@github.com:
2023-01-11T21:15:40.5141569Z ##[endgroup]
2023-01-11T21:15:40.5142187Z ##[group]Fetching submodules
2023-01-11T21:15:40.5147315Z [command]/usr/bin/git submodule sync --recursive
2023-01-11T21:15:40.5476431Z [command]/usr/bin/git -c protocol.version=2 submodule update --init --force --recursive
2023-01-11T21:15:40.5787376Z Submodule 'android/libs/fbjni' (https://github.com/facebookincubator/fbjni.git) registered for path 'android/libs/fbjni'
2023-01-11T21:15:40.5790647Z Submodule 'third_party/NNPACK_deps/FP16' (https://github.com/Maratyszcza/FP16.git) registered for path 'third_party/FP16'
2023-01-11T21:15:40.5794133Z Submodule 'third_party/NNPACK_deps/FXdiv' (https://github.com/Maratyszcza/FXdiv.git) registered for path 'third_party/FXdiv'
2023-01-11T21:15:40.5797755Z Submodule 'third_party/NNPACK' (https://github.com/Maratyszcza/NNPACK.git) registered for path 'third_party/NNPACK'
2023-01-11T21:15:40.5801605Z Submodule 'third_party/QNNPACK' (https://github.com/pytorch/QNNPACK) registered for path 'third_party/QNNPACK'
2023-01-11T21:15:40.5805753Z Submodule 'third_party/VulkanMemoryAllocator' (https://github.com/GPUOpen-LibrariesAndSDKs/VulkanMemoryAllocator.git) registered for path 'third_party/VulkanMemoryAllocator'
2023-01-11T21:15:40.5810170Z Submodule 'third_party/XNNPACK' (https://github.com/google/XNNPACK.git) registered for path 'third_party/XNNPACK'
2023-01-11T21:15:40.5814331Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/benchmark'
2023-01-11T21:15:40.5818746Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo.git) registered for path 'third_party/cpuinfo'
2023-01-11T21:15:40.5823324Z Submodule 'third_party/cub' (https://github.com/NVlabs/cub.git) registered for path 'third_party/cub'
2023-01-11T21:15:40.5828152Z Submodule 'third_party/cudnn_frontend' (https://github.com/NVIDIA/cudnn-frontend.git) registered for path 'third_party/cudnn_frontend'
2023-01-11T21:15:40.5832826Z Submodule 'third_party/cutlass' (https://github.com/NVIDIA/cutlass.git) registered for path 'third_party/cutlass'
2023-01-11T21:15:40.5837792Z Submodule 'third_party/eigen' (https://gitlab.com/libeigen/eigen.git) registered for path 'third_party/eigen'
2023-01-11T21:15:40.5843149Z Submodule 'third_party/fbgemm' (https://github.com/pytorch/fbgemm) registered for path 'third_party/fbgemm'
2023-01-11T21:15:40.5848614Z Submodule 'third_party/flatbuffers' (https://github.com/google/flatbuffers.git) registered for path 'third_party/flatbuffers'
2023-01-11T21:15:40.5853867Z Submodule 'third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/fmt'
2023-01-11T21:15:40.5859322Z Submodule 'third_party/foxi' (https://github.com/houseroad/foxi.git) registered for path 'third_party/foxi'
2023-01-11T21:15:40.5864959Z Submodule 'third_party/gemmlowp/gemmlowp' (https://github.com/google/gemmlowp.git) registered for path 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:15:40.5870583Z Submodule 'third_party/gloo' (https://github.com/facebookincubator/gloo) registered for path 'third_party/gloo'
2023-01-11T21:15:40.5876625Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/googletest'
2023-01-11T21:15:40.5882585Z Submodule 'third_party/ideep' (https://github.com/intel/ideep) registered for path 'third_party/ideep'
2023-01-11T21:15:40.5889329Z Submodule 'third_party/ios-cmake' (https://github.com/Yangqing/ios-cmake.git) registered for path 'third_party/ios-cmake'
2023-01-11T21:15:40.5895510Z Submodule 'third_party/ittapi' (https://github.com/intel/ittapi.git) registered for path 'third_party/ittapi'
2023-01-11T21:15:40.5901912Z Submodule 'third_party/kineto' (https://github.com/pytorch/kineto) registered for path 'third_party/kineto'
2023-01-11T21:15:40.5908398Z Submodule 'third_party/nccl/nccl' (https://github.com/NVIDIA/nccl) registered for path 'third_party/nccl/nccl'
2023-01-11T21:15:40.5915184Z Submodule 'third_party/neon2sse' (https://github.com/intel/ARM_NEON_2_x86_SSE.git) registered for path 'third_party/neon2sse'
2023-01-11T21:15:40.5921843Z Submodule 'third_party/nlohmann' (https://github.com/nlohmann/json.git) registered for path 'third_party/nlohmann'
2023-01-11T21:15:40.5929078Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx'
2023-01-11T21:15:40.5936300Z Submodule 'third_party/onnx-tensorrt' (https://github.com/onnx/onnx-tensorrt) registered for path 'third_party/onnx-tensorrt'
2023-01-11T21:15:40.5943415Z Submodule 'third_party/pocketfft' (https://github.com/mreineck/pocketfft) registered for path 'third_party/pocketfft'
2023-01-11T21:15:40.5950837Z Submodule 'third_party/protobuf' (https://github.com/protocolbuffers/protobuf.git) registered for path 'third_party/protobuf'
2023-01-11T21:15:40.5958230Z Submodule 'third_party/NNPACK_deps/psimd' (https://github.com/Maratyszcza/psimd.git) registered for path 'third_party/psimd'
2023-01-11T21:15:40.5966187Z Submodule 'third_party/NNPACK_deps/pthreadpool' (https://github.com/Maratyszcza/pthreadpool.git) registered for path 'third_party/pthreadpool'
2023-01-11T21:15:40.5973812Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/pybind11'
2023-01-11T21:15:40.5981873Z Submodule 'third_party/python-enum' (https://github.com/PeachPy/enum34.git) registered for path 'third_party/python-enum'
2023-01-11T21:15:40.5989684Z Submodule 'third_party/python-peachpy' (https://github.com/malfet/PeachPy.git) registered for path 'third_party/python-peachpy'
2023-01-11T21:15:40.5997733Z Submodule 'third_party/python-six' (https://github.com/benjaminp/six.git) registered for path 'third_party/python-six'
2023-01-11T21:15:40.6006245Z Submodule 'third_party/sleef' (https://github.com/shibatch/sleef) registered for path 'third_party/sleef'
2023-01-11T21:15:40.6014816Z Submodule 'third_party/tbb' (https://github.com/01org/tbb) registered for path 'third_party/tbb'
2023-01-11T21:15:40.6023358Z Submodule 'third_party/tensorpipe' (https://github.com/pytorch/tensorpipe.git) registered for path 'third_party/tensorpipe'
2023-01-11T21:15:40.6031929Z Submodule 'third_party/zstd' (https://github.com/facebook/zstd.git) registered for path 'third_party/zstd'
2023-01-11T21:15:40.6062992Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/android/libs/fbjni'...
2023-01-11T21:15:40.9327842Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FP16'...
2023-01-11T21:15:41.1706238Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/FXdiv'...
2023-01-11T21:15:41.4114618Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/NNPACK'...
2023-01-11T21:15:41.7256531Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/QNNPACK'...
2023-01-11T21:15:42.0260001Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/VulkanMemoryAllocator'...
2023-01-11T21:15:44.2482501Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/XNNPACK'...
2023-01-11T21:15:50.2687759Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/benchmark'...
2023-01-11T21:15:50.7104523Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cpuinfo'...
2023-01-11T21:15:51.2818673Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cub'...
2023-01-11T21:15:52.8663864Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cudnn_frontend'...
2023-01-11T21:15:54.2492449Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/cutlass'...
2023-01-11T21:15:55.6584562Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/eigen'...
2023-01-11T21:16:02.7514459Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm'...
2023-01-11T21:16:03.7270099Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/flatbuffers'...
2023-01-11T21:16:06.5988086Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fmt'...
2023-01-11T21:16:07.7796607Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/foxi'...
2023-01-11T21:16:08.0125547Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gemmlowp/gemmlowp'...
2023-01-11T21:16:08.5199935Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/gloo'...
2023-01-11T21:16:09.2448024Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/googletest'...
2023-01-11T21:16:10.4826917Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep'...
2023-01-11T21:16:12.0835650Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ios-cmake'...
2023-01-11T21:16:12.3052732Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ittapi'...
2023-01-11T21:16:12.6118938Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto'...
2023-01-11T21:16:14.7870152Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nccl/nccl'...
2023-01-11T21:16:15.1449372Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/neon2sse'...
2023-01-11T21:16:15.5961023Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/nlohmann'...
2023-01-11T21:16:21.7130898Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx'...
2023-01-11T21:16:23.5151306Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt'...
2023-01-11T21:16:23.9859817Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pocketfft'...
2023-01-11T21:16:24.2586482Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf'...
2023-01-11T21:16:30.7124598Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/psimd'...
2023-01-11T21:16:30.9178897Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pthreadpool'...
2023-01-11T21:16:31.1880151Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/pybind11'...
2023-01-11T21:16:32.1557718Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-enum'...
2023-01-11T21:16:32.4367647Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-peachpy'...
2023-01-11T21:16:32.7720199Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/python-six'...
2023-01-11T21:16:33.1321710Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/sleef'...
2023-01-11T21:16:33.7515450Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tbb'...
2023-01-11T21:16:36.0195991Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe'...
2023-01-11T21:16:36.5542275Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/zstd'...
2023-01-11T21:16:39.0618987Z Submodule path 'android/libs/fbjni': checked out '7e1e1fe3858c63c251c637ae41a20de425dde96f'
2023-01-11T21:16:39.0746778Z Submodule path 'third_party/FP16': checked out '4dfe081cf6bcd15db339cf2680b9281b8451eeb3'
2023-01-11T21:16:39.0847133Z Submodule path 'third_party/FXdiv': checked out 'b408327ac2a15ec3e43352421954f5b1967701d1'
2023-01-11T21:16:39.1134146Z Submodule path 'third_party/NNPACK': checked out 'c07e3a0400713d546e0dea2d5466dd22ea389c73'
2023-01-11T21:16:39.1409621Z Submodule path 'third_party/QNNPACK': checked out '7d2a4e9931a82adc3814275b6219a03e24e36b4c'
2023-01-11T21:16:39.1869683Z Submodule path 'third_party/VulkanMemoryAllocator': checked out 'a6bfc237255a6bac1513f7c1ebde6d8aed6b5191'
2023-01-11T21:16:39.9823330Z Submodule path 'third_party/XNNPACK': checked out 'ae108ef49aa5623b896fc93d4298c49d1750d9ba'
2023-01-11T21:16:40.0075903Z Submodule path 'third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2023-01-11T21:16:40.1597335Z Submodule path 'third_party/cpuinfo': checked out '8ec7bd91ad0470e61cf38f618cc1f270dede599c'
2023-01-11T21:16:40.2027063Z Submodule path 'third_party/cub': checked out 'd106ddb991a56c3df1b6d51b2409e36ba8181ce4'
2023-01-11T21:16:40.5696968Z Submodule path 'third_party/cudnn_frontend': checked out '171a7a986f7fbd9ed71bd0cf3c7ad4f55843d6b3'
2023-01-11T21:16:41.0988931Z Submodule path 'third_party/cutlass': checked out 'b72cbf957df8cf84a6d0ff91c190ad51a9c1d24a'
2023-01-11T21:16:41.4047223Z Submodule path 'third_party/eigen': checked out '3147391d946bb4b6c68edd901f2add6ac1f31f8c'
2023-01-11T21:16:41.4616092Z Submodule path 'third_party/fbgemm': checked out '80d64206c07879fd4683be66873de7cefa1a0a71'
2023-01-11T21:16:41.4634402Z Submodule 'third_party/asmjit' (https://github.com/asmjit/asmjit.git) registered for path 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:16:41.4637080Z Submodule 'third_party/cpuinfo' (https://github.com/pytorch/cpuinfo) registered for path 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:16:41.4640286Z Submodule 'third_party/googletest' (https://github.com/google/googletest) registered for path 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:16:41.4643818Z Submodule 'third_party/hipify_torch' (https://github.com/ROCmSoftwarePlatform/hipify_torch.git) registered for path 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:16:41.4670772Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/asmjit'...
2023-01-11T21:16:42.4715801Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/cpuinfo'...
2023-01-11T21:16:43.0455902Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/googletest'...
2023-01-11T21:16:44.0573872Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/fbgemm/third_party/hipify_torch'...
2023-01-11T21:16:44.4208508Z Submodule path 'third_party/fbgemm/third_party/asmjit': checked out 'd3fbf7c9bc7c1d1365a94a45614b91c5a3706b81'
2023-01-11T21:16:44.5459229Z Submodule path 'third_party/fbgemm/third_party/cpuinfo': checked out 'ed8b86a253800bafdb7b25c5c399f91bff9cb1f3'
2023-01-11T21:16:44.6163829Z Submodule path 'third_party/fbgemm/third_party/googletest': checked out 'cbf019de22c8dd37b2108da35b2748fd702d1796'
2023-01-11T21:16:44.6281381Z Submodule path 'third_party/fbgemm/third_party/hipify_torch': checked out '1840658c184f3eeba787dae0f06c45756c1daaf5'
2023-01-11T21:16:44.7364638Z Submodule path 'third_party/flatbuffers': checked out 'd0cede9c90c5257537c293517a21376408b549fa'
2023-01-11T21:16:44.7801412Z Submodule path 'third_party/fmt': checked out '7bdf0628b1276379886c7f6dda2cef2b3b374f0b'
2023-01-11T21:16:44.7903392Z Submodule path 'third_party/foxi': checked out 'c278588e34e535f0bb8f00df3880d26928038cad'
2023-01-11T21:16:44.8388312Z Submodule path 'third_party/gemmlowp/gemmlowp': checked out '3fb5c176c17c765a3492cd2f0321b0dab712f350'
2023-01-11T21:16:44.8673115Z Submodule path 'third_party/gloo': checked out '4a5e339b764261d20fc409071dc7a8b8989aa195'
2023-01-11T21:16:44.9218253Z Submodule path 'third_party/googletest': checked out 'e2239ee6043f73722e7aa812a459f54a28552929'
2023-01-11T21:16:44.9351119Z Submodule path 'third_party/ideep': checked out 'e533c771a1e75a1c225c14b2261eefa62681d9e6'
2023-01-11T21:16:44.9369647Z Submodule 'mkl-dnn' (https://github.com/intel/mkl-dnn.git) registered for path 'third_party/ideep/mkl-dnn'
2023-01-11T21:16:44.9397029Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn'...
2023-01-11T21:16:53.9475726Z Submodule path 'third_party/ideep/mkl-dnn': checked out '404ad76ee633c939d705eb583ffe50a806969d5e'
2023-01-11T21:16:53.9496402Z Submodule 'third_party/oneDNN' (https://github.com/oneapi-src/oneDNN.git) registered for path 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:16:53.9524617Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/ideep/mkl-dnn/third_party/oneDNN'...
2023-01-11T21:17:03.1370317Z Submodule path 'third_party/ideep/mkl-dnn/third_party/oneDNN': checked out 'fbec3e25a559ee252022ae066817b204e106a6ba'
2023-01-11T21:17:03.1487629Z Submodule path 'third_party/ios-cmake': checked out '8abaed637d56f1337d6e1d2c4026e25c1eade724'
2023-01-11T21:17:03.1663006Z Submodule path 'third_party/ittapi': checked out '5b8a7d7422611c3a0d799fb5fc5dd4abfae35b42'
2023-01-11T21:17:03.2786230Z Submodule path 'third_party/kineto': checked out '6c1629809068efd78a8d56b4aa479c7ec49ae562'
2023-01-11T21:17:03.2804418Z Submodule 'libkineto/third_party/fmt' (https://github.com/fmtlib/fmt.git) registered for path 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:17:03.2808017Z Submodule 'libkineto/third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:17:03.2835626Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/fmt'...
2023-01-11T21:17:04.5055364Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/kineto/libkineto/third_party/googletest'...
2023-01-11T21:17:05.5888594Z Submodule path 'third_party/kineto/libkineto/third_party/fmt': checked out '2591ab91c3898c9f6544fff04660276537d32ffd'
2023-01-11T21:17:05.6536912Z Submodule path 'third_party/kineto/libkineto/third_party/googletest': checked out '7aca84427f224eeed3144123d5230d5871e93347'
2023-01-11T21:17:05.6790345Z Submodule path 'third_party/nccl/nccl': checked out 'f89fd4777d2ef9229c039ff750ae21da01626f52'
2023-01-11T21:17:05.6952904Z Submodule path 'third_party/neon2sse': checked out '97a126f08ce318023be604d03f88bf0820a9464a'
2023-01-11T21:17:05.8273264Z Submodule path 'third_party/nlohmann': checked out '87cda1d6646592ac5866dc703c8e1839046a6806'
2023-01-11T21:17:06.1582909Z Submodule path 'third_party/onnx': checked out 'f7ee1ac60d06abe8e26c9b6bbe1e3db5286b614b'
2023-01-11T21:17:06.1614873Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx/third_party/benchmark'
2023-01-11T21:17:06.1617991Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx/third_party/pybind11'
2023-01-11T21:17:06.1646425Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/benchmark'...
2023-01-11T21:17:06.6468804Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx/third_party/pybind11'...
2023-01-11T21:17:07.6315374Z Submodule path 'third_party/onnx/third_party/benchmark': checked out '0d98dba29d66e93259db7daa53a9327df767a415'
2023-01-11T21:17:07.6690548Z Submodule path 'third_party/onnx/third_party/pybind11': checked out 'ffa346860b306c9bbfb341aed9c14c067751feb8'
2023-01-11T21:17:07.6866799Z Submodule path 'third_party/onnx-tensorrt': checked out 'c153211418a7c57ce071d9ce2a41f8d1c85a878f'
2023-01-11T21:17:07.6883694Z Submodule 'third_party/onnx' (https://github.com/onnx/onnx.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:17:07.6910961Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx'...
2023-01-11T21:17:09.7053871Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx': checked out '765f5ee823a67a866f4bd28a9860e81f3c811ce8'
2023-01-11T21:17:09.7076152Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:17:09.7079290Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:17:09.7107624Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'...
2023-01-11T21:17:10.1806765Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'...
2023-01-11T21:17:11.1365155Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark': checked out 'e776aa0275e293707b6a0901e0e8d8a8a3679508'
2023-01-11T21:17:11.2163098Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11': checked out 'a1041190c8b8ff0cd9e2f0752248ad5e3789ea0c'
2023-01-11T21:17:11.2180209Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:17:11.2207918Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'...
2023-01-11T21:17:11.4707441Z Submodule path 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2023-01-11T21:17:11.4814896Z Submodule path 'third_party/pocketfft': checked out 'ea778e37710c07723435b1be58235996d1d43a5a'
2023-01-11T21:17:11.8017093Z Submodule path 'third_party/protobuf': checked out 'd1eca4e4b421cd2997495c4b4e65cea6be4e9b8a'
2023-01-11T21:17:11.8040014Z Submodule 'third_party/benchmark' (https://github.com/google/benchmark.git) registered for path 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:17:11.8043797Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/protobuf/third_party/googletest'
2023-01-11T21:17:11.8071585Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/benchmark'...
2023-01-11T21:17:12.2862277Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/protobuf/third_party/googletest'...
2023-01-11T21:17:13.2806114Z Submodule path 'third_party/protobuf/third_party/benchmark': checked out '5b7683f49e1e9223cf9927b24f6fd3d6bd82e3f8'
2023-01-11T21:17:13.3626366Z Submodule path 'third_party/protobuf/third_party/googletest': checked out '5ec7f0c4a113e2f18ac2c6cc7df51ad6afc24081'
2023-01-11T21:17:13.3720506Z Submodule path 'third_party/psimd': checked out '072586a71b55b7f8c584153d223e95687148a900'
2023-01-11T21:17:13.3847254Z Submodule path 'third_party/pthreadpool': checked out 'a134dd5d4cee80cce15db81a72e7f929d71dd413'
2023-01-11T21:17:13.4241558Z Submodule path 'third_party/pybind11': checked out '80dc998efced8ceb2be59756668a7e90e8bef917'
2023-01-11T21:17:13.4341439Z Submodule path 'third_party/python-enum': checked out '4cfedc426c4e2fc52e3f5c2b4297e15ed8d6b8c7'
2023-01-11T21:17:13.4682708Z Submodule path 'third_party/python-peachpy': checked out 'f45429b087dd7d5bc78bb40dc7cf06425c252d67'
2023-01-11T21:17:13.4785982Z Submodule path 'third_party/python-six': checked out '15e31431af97e5e64b80af0a3f598d382bcdd49a'
2023-01-11T21:17:13.5302079Z Submodule path 'third_party/sleef': checked out 'e0a003ee838b75d11763aa9c3ef17bf71a725bff'
2023-01-11T21:17:13.6649346Z Submodule path 'third_party/tbb': checked out 'a51a90bc609bb73db8ea13841b5cf7aa4344d4a9'
2023-01-11T21:17:13.6986193Z Submodule path 'third_party/tensorpipe': checked out '52791a2fd214b2a9dc5759d36725909c1daa7f2e'
2023-01-11T21:17:13.7004179Z Submodule 'third_party/googletest' (https://github.com/google/googletest.git) registered for path 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:17:13.7010378Z Submodule 'third_party/libnop' (https://github.com/google/libnop.git) registered for path 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:17:13.7014141Z Submodule 'third_party/libuv' (https://github.com/libuv/libuv.git) registered for path 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:17:13.7017268Z Submodule 'third_party/pybind11' (https://github.com/pybind/pybind11.git) registered for path 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:17:13.7044134Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/googletest'...
2023-01-11T21:17:16.2062846Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libnop'...
2023-01-11T21:17:16.4942666Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/libuv'...
2023-01-11T21:17:17.7620172Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11'...
2023-01-11T21:17:18.7948587Z Submodule path 'third_party/tensorpipe/third_party/googletest': checked out 'aee0f9d9b5b87796ee8a0ab26b7587ec30e8858e'
2023-01-11T21:17:18.8120712Z Submodule path 'third_party/tensorpipe/third_party/libnop': checked out '910b55815be16109f04f4180e9adee14fb4ce281'
2023-01-11T21:17:18.8889978Z Submodule path 'third_party/tensorpipe/third_party/libuv': checked out '1dff88e5161cba5c59276d2070d2e304e4dcb242'
2023-01-11T21:17:18.9210637Z Submodule path 'third_party/tensorpipe/third_party/pybind11': checked out 'a23996fce38ff6ccfbcdc09f1e63f2c4be5ea2ef'
2023-01-11T21:17:18.9227603Z Submodule 'tools/clang' (https://github.com/wjakob/clang-cindex-python3) registered for path 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:17:18.9255371Z Cloning into '/home/ec2-user/actions-runner/_work/pytorch/pytorch/third_party/tensorpipe/third_party/pybind11/tools/clang'...
2023-01-11T21:17:19.1792876Z Submodule path 'third_party/tensorpipe/third_party/pybind11/tools/clang': checked out '6a00cbc4a9b8e68b71caf7f774b3f9c753ae84d5'
2023-01-11T21:17:19.3438891Z Submodule path 'third_party/zstd': checked out 'aec56a52fbab207fc639a1937d1e708a282edca8'
2023-01-11T21:17:19.3471396Z [command]/usr/bin/git submodule foreach --recursive git config --local gc.auto 0
2023-01-11T21:17:19.3801151Z Entering 'android/libs/fbjni'
2023-01-11T21:17:19.3845855Z Entering 'third_party/FP16'
2023-01-11T21:17:19.3890327Z Entering 'third_party/FXdiv'
2023-01-11T21:17:19.3934047Z Entering 'third_party/NNPACK'
2023-01-11T21:17:19.3978610Z Entering 'third_party/QNNPACK'
2023-01-11T21:17:19.4022695Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T21:17:19.4068433Z Entering 'third_party/XNNPACK'
2023-01-11T21:17:19.4123438Z Entering 'third_party/benchmark'
2023-01-11T21:17:19.4167833Z Entering 'third_party/cpuinfo'
2023-01-11T21:17:19.4213553Z Entering 'third_party/cub'
2023-01-11T21:17:19.4257509Z Entering 'third_party/cudnn_frontend'
2023-01-11T21:17:19.4308144Z Entering 'third_party/cutlass'
2023-01-11T21:17:19.4359784Z Entering 'third_party/eigen'
2023-01-11T21:17:19.4406724Z Entering 'third_party/fbgemm'
2023-01-11T21:17:19.4451190Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:17:19.4495037Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:17:19.4540020Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:17:19.4583682Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:17:19.4628362Z Entering 'third_party/flatbuffers'
2023-01-11T21:17:19.4676339Z Entering 'third_party/fmt'
2023-01-11T21:17:19.4720262Z Entering 'third_party/foxi'
2023-01-11T21:17:19.4765313Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:17:19.4810148Z Entering 'third_party/gloo'
2023-01-11T21:17:19.4854888Z Entering 'third_party/googletest'
2023-01-11T21:17:19.4899148Z Entering 'third_party/ideep'
2023-01-11T21:17:19.4942092Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T21:17:19.4987927Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:17:19.5038527Z Entering 'third_party/ios-cmake'
2023-01-11T21:17:19.5082571Z Entering 'third_party/ittapi'
2023-01-11T21:17:19.5126580Z Entering 'third_party/kineto'
2023-01-11T21:17:19.5170886Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:17:19.5215032Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:17:19.5260054Z Entering 'third_party/nccl/nccl'
2023-01-11T21:17:19.5306774Z Entering 'third_party/neon2sse'
2023-01-11T21:17:19.5350361Z Entering 'third_party/nlohmann'
2023-01-11T21:17:19.5395400Z Entering 'third_party/onnx'
2023-01-11T21:17:19.5454602Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T21:17:19.5498196Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T21:17:19.5543947Z Entering 'third_party/onnx-tensorrt'
2023-01-11T21:17:19.5587305Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:17:19.5636690Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:17:19.5681105Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:17:19.5726214Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:17:19.5775491Z Entering 'third_party/pocketfft'
2023-01-11T21:17:19.5819115Z Entering 'third_party/protobuf'
2023-01-11T21:17:19.5867539Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:17:19.5911725Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T21:17:19.5957132Z Entering 'third_party/psimd'
2023-01-11T21:17:19.6001676Z Entering 'third_party/pthreadpool'
2023-01-11T21:17:19.6045857Z Entering 'third_party/pybind11'
2023-01-11T21:17:19.6090980Z Entering 'third_party/python-enum'
2023-01-11T21:17:19.6134903Z Entering 'third_party/python-peachpy'
2023-01-11T21:17:19.6178609Z Entering 'third_party/python-six'
2023-01-11T21:17:19.6223041Z Entering 'third_party/sleef'
2023-01-11T21:17:19.6267122Z Entering 'third_party/tbb'
2023-01-11T21:17:19.6314182Z Entering 'third_party/tensorpipe'
2023-01-11T21:17:19.6358573Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:17:19.6402817Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:17:19.6446732Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:17:19.6491482Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:17:19.6533793Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:17:19.6579621Z Entering 'third_party/zstd'
2023-01-11T21:17:19.6635187Z ##[endgroup]
2023-01-11T21:17:19.6637951Z ##[group]Persisting credentials for submodules
2023-01-11T21:17:19.6643139Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'url\.https\:\/\/github\.com\/\.insteadOf' && git config --local --unset-all 'url.https://github.com/.insteadOf' || :
2023-01-11T21:17:19.6968340Z Entering 'android/libs/fbjni'
2023-01-11T21:17:19.7011615Z Entering 'third_party/FP16'
2023-01-11T21:17:19.7055244Z Entering 'third_party/FXdiv'
2023-01-11T21:17:19.7099160Z Entering 'third_party/NNPACK'
2023-01-11T21:17:19.7142705Z Entering 'third_party/QNNPACK'
2023-01-11T21:17:19.7186969Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T21:17:19.7230921Z Entering 'third_party/XNNPACK'
2023-01-11T21:17:19.7286572Z Entering 'third_party/benchmark'
2023-01-11T21:17:19.7330306Z Entering 'third_party/cpuinfo'
2023-01-11T21:17:19.7374677Z Entering 'third_party/cub'
2023-01-11T21:17:19.7417521Z Entering 'third_party/cudnn_frontend'
2023-01-11T21:17:19.7467227Z Entering 'third_party/cutlass'
2023-01-11T21:17:19.7518130Z Entering 'third_party/eigen'
2023-01-11T21:17:19.7565263Z Entering 'third_party/fbgemm'
2023-01-11T21:17:19.7608784Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:17:19.7652527Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:17:19.7694952Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:17:19.7737533Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:17:19.7781178Z Entering 'third_party/flatbuffers'
2023-01-11T21:17:19.7827383Z Entering 'third_party/fmt'
2023-01-11T21:17:19.7871036Z Entering 'third_party/foxi'
2023-01-11T21:17:19.7914225Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:17:19.7958687Z Entering 'third_party/gloo'
2023-01-11T21:17:19.8002069Z Entering 'third_party/googletest'
2023-01-11T21:17:19.8045741Z Entering 'third_party/ideep'
2023-01-11T21:17:19.8087744Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T21:17:19.8134688Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:17:19.8184944Z Entering 'third_party/ios-cmake'
2023-01-11T21:17:19.8228161Z Entering 'third_party/ittapi'
2023-01-11T21:17:19.8271696Z Entering 'third_party/kineto'
2023-01-11T21:17:19.8314678Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:17:19.8357821Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:17:19.8402030Z Entering 'third_party/nccl/nccl'
2023-01-11T21:17:19.8445850Z Entering 'third_party/neon2sse'
2023-01-11T21:17:19.8489107Z Entering 'third_party/nlohmann'
2023-01-11T21:17:19.8533934Z Entering 'third_party/onnx'
2023-01-11T21:17:19.8590824Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T21:17:19.8634883Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T21:17:19.8680167Z Entering 'third_party/onnx-tensorrt'
2023-01-11T21:17:19.8722521Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:17:19.8770737Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:17:19.8814705Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:17:19.8858702Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:17:19.8906127Z Entering 'third_party/pocketfft'
2023-01-11T21:17:19.8948704Z Entering 'third_party/protobuf'
2023-01-11T21:17:19.8995874Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:17:19.9038920Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T21:17:19.9083051Z Entering 'third_party/psimd'
2023-01-11T21:17:19.9126882Z Entering 'third_party/pthreadpool'
2023-01-11T21:17:19.9170064Z Entering 'third_party/pybind11'
2023-01-11T21:17:19.9213405Z Entering 'third_party/python-enum'
2023-01-11T21:17:19.9257862Z Entering 'third_party/python-peachpy'
2023-01-11T21:17:19.9301043Z Entering 'third_party/python-six'
2023-01-11T21:17:19.9344088Z Entering 'third_party/sleef'
2023-01-11T21:17:19.9388225Z Entering 'third_party/tbb'
2023-01-11T21:17:19.9433121Z Entering 'third_party/tensorpipe'
2023-01-11T21:17:19.9476585Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:17:19.9519640Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:17:19.9562016Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:17:19.9605239Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:17:19.9647251Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:17:19.9693629Z Entering 'third_party/zstd'
2023-01-11T21:17:19.9751058Z [command]/usr/bin/git submodule foreach --recursive git config --local 'http.https://github.com/.extraheader' 'AUTHORIZATION: basic ***' && git config --local --show-origin --name-only --get-regexp remote.origin.url
2023-01-11T21:17:20.0076829Z Entering 'android/libs/fbjni'
2023-01-11T21:17:20.0117055Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/android/libs/fbjni/config	remote.origin.url
2023-01-11T21:17:20.0135802Z Entering 'third_party/FP16'
2023-01-11T21:17:20.0175961Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FP16/config	remote.origin.url
2023-01-11T21:17:20.0193393Z Entering 'third_party/FXdiv'
2023-01-11T21:17:20.0234203Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/FXdiv/config	remote.origin.url
2023-01-11T21:17:20.0252778Z Entering 'third_party/NNPACK'
2023-01-11T21:17:20.0293024Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK/config	remote.origin.url
2023-01-11T21:17:20.0311369Z Entering 'third_party/QNNPACK'
2023-01-11T21:17:20.0351542Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/QNNPACK/config	remote.origin.url
2023-01-11T21:17:20.0370438Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T21:17:20.0410336Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/VulkanMemoryAllocator/config	remote.origin.url
2023-01-11T21:17:20.0428438Z Entering 'third_party/XNNPACK'
2023-01-11T21:17:20.0470047Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/XNNPACK/config	remote.origin.url
2023-01-11T21:17:20.0502394Z Entering 'third_party/benchmark'
2023-01-11T21:17:20.0543307Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T21:17:20.0562454Z Entering 'third_party/cpuinfo'
2023-01-11T21:17:20.0605446Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cpuinfo/config	remote.origin.url
2023-01-11T21:17:20.0625638Z Entering 'third_party/cub'
2023-01-11T21:17:20.0666475Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cub/config	remote.origin.url
2023-01-11T21:17:20.0687045Z Entering 'third_party/cudnn_frontend'
2023-01-11T21:17:20.0729343Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cudnn_frontend/config	remote.origin.url
2023-01-11T21:17:20.0753709Z Entering 'third_party/cutlass'
2023-01-11T21:17:20.0794226Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/cutlass/config	remote.origin.url
2023-01-11T21:17:20.0820314Z Entering 'third_party/eigen'
2023-01-11T21:17:20.0860505Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/eigen/config	remote.origin.url
2023-01-11T21:17:20.0882254Z Entering 'third_party/fbgemm'
2023-01-11T21:17:20.0923134Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/config	remote.origin.url
2023-01-11T21:17:20.0941977Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:17:20.0985250Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/asmjit/config	remote.origin.url
2023-01-11T21:17:20.1002650Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:17:20.1042992Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/cpuinfo/config	remote.origin.url
2023-01-11T21:17:20.1107408Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:17:20.1148307Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/googletest/config	remote.origin.url
2023-01-11T21:17:20.1166818Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:17:20.1207401Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fbgemm/modules/third_party/hipify_torch/config	remote.origin.url
2023-01-11T21:17:20.1226643Z Entering 'third_party/flatbuffers'
2023-01-11T21:17:20.1269214Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/flatbuffers/config	remote.origin.url
2023-01-11T21:17:20.1292628Z Entering 'third_party/fmt'
2023-01-11T21:17:20.1331987Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/fmt/config	remote.origin.url
2023-01-11T21:17:20.1350836Z Entering 'third_party/foxi'
2023-01-11T21:17:20.1393672Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/foxi/config	remote.origin.url
2023-01-11T21:17:20.1414080Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:17:20.1455845Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gemmlowp/gemmlowp/config	remote.origin.url
2023-01-11T21:17:20.1474884Z Entering 'third_party/gloo'
2023-01-11T21:17:20.1516377Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/gloo/config	remote.origin.url
2023-01-11T21:17:20.1536251Z Entering 'third_party/googletest'
2023-01-11T21:17:20.1578207Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/googletest/config	remote.origin.url
2023-01-11T21:17:20.1597850Z Entering 'third_party/ideep'
2023-01-11T21:17:20.1640300Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/config	remote.origin.url
2023-01-11T21:17:20.1661193Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T21:17:20.1703268Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/config	remote.origin.url
2023-01-11T21:17:20.1723966Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:17:20.1766443Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ideep/modules/mkl-dnn/modules/third_party/oneDNN/config	remote.origin.url
2023-01-11T21:17:20.1793367Z Entering 'third_party/ios-cmake'
2023-01-11T21:17:20.1835353Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ios-cmake/config	remote.origin.url
2023-01-11T21:17:20.1855136Z Entering 'third_party/ittapi'
2023-01-11T21:17:20.1896664Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/ittapi/config	remote.origin.url
2023-01-11T21:17:20.1915896Z Entering 'third_party/kineto'
2023-01-11T21:17:20.1958812Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/config	remote.origin.url
2023-01-11T21:17:20.1978924Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:17:20.2021508Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/fmt/config	remote.origin.url
2023-01-11T21:17:20.2039975Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:17:20.2081829Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/kineto/modules/libkineto/third_party/googletest/config	remote.origin.url
2023-01-11T21:17:20.2103055Z Entering 'third_party/nccl/nccl'
2023-01-11T21:17:20.2145156Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nccl/nccl/config	remote.origin.url
2023-01-11T21:17:20.2164334Z Entering 'third_party/neon2sse'
2023-01-11T21:17:20.2203802Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/neon2sse/config	remote.origin.url
2023-01-11T21:17:20.2222281Z Entering 'third_party/nlohmann'
2023-01-11T21:17:20.2263576Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/nlohmann/config	remote.origin.url
2023-01-11T21:17:20.2282589Z Entering 'third_party/onnx'
2023-01-11T21:17:20.2323491Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/config	remote.origin.url
2023-01-11T21:17:20.2356411Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T21:17:20.2397504Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T21:17:20.2416109Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T21:17:20.2459547Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T21:17:20.2480460Z Entering 'third_party/onnx-tensorrt'
2023-01-11T21:17:20.2522926Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/config	remote.origin.url
2023-01-11T21:17:20.2541549Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:17:20.2582002Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/config	remote.origin.url
2023-01-11T21:17:20.2606444Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:17:20.2647510Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T21:17:20.2666034Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:17:20.2707340Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T21:17:20.2724600Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:17:20.2765416Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/onnx-tensorrt/modules/third_party/onnx/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2023-01-11T21:17:20.2788104Z Entering 'third_party/pocketfft'
2023-01-11T21:17:20.2829104Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pocketfft/config	remote.origin.url
2023-01-11T21:17:20.2847089Z Entering 'third_party/protobuf'
2023-01-11T21:17:20.2887465Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/config	remote.origin.url
2023-01-11T21:17:20.2909638Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:17:20.2950170Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/benchmark/config	remote.origin.url
2023-01-11T21:17:20.2968787Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T21:17:20.3009232Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/protobuf/modules/third_party/googletest/config	remote.origin.url
2023-01-11T21:17:20.3029585Z Entering 'third_party/psimd'
2023-01-11T21:17:20.3072385Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/psimd/config	remote.origin.url
2023-01-11T21:17:20.3091931Z Entering 'third_party/pthreadpool'
2023-01-11T21:17:20.3133245Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/NNPACK_deps/pthreadpool/config	remote.origin.url
2023-01-11T21:17:20.3151530Z Entering 'third_party/pybind11'
2023-01-11T21:17:20.3194620Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T21:17:20.3214823Z Entering 'third_party/python-enum'
2023-01-11T21:17:20.3256725Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-enum/config	remote.origin.url
2023-01-11T21:17:20.3276620Z Entering 'third_party/python-peachpy'
2023-01-11T21:17:20.3318769Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-peachpy/config	remote.origin.url
2023-01-11T21:17:20.3337607Z Entering 'third_party/python-six'
2023-01-11T21:17:20.3379046Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/python-six/config	remote.origin.url
2023-01-11T21:17:20.3396954Z Entering 'third_party/sleef'
2023-01-11T21:17:20.3439586Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/sleef/config	remote.origin.url
2023-01-11T21:17:20.3461072Z Entering 'third_party/tbb'
2023-01-11T21:17:20.3504224Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tbb/config	remote.origin.url
2023-01-11T21:17:20.3526237Z Entering 'third_party/tensorpipe'
2023-01-11T21:17:20.3567255Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/config	remote.origin.url
2023-01-11T21:17:20.3585636Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:17:20.3627256Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/googletest/config	remote.origin.url
2023-01-11T21:17:20.3646353Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:17:20.3687650Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libnop/config	remote.origin.url
2023-01-11T21:17:20.3705400Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:17:20.3746411Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/libuv/config	remote.origin.url
2023-01-11T21:17:20.3764170Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:17:20.3811161Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/config	remote.origin.url
2023-01-11T21:17:20.3831235Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:17:20.3872377Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/tensorpipe/modules/third_party/pybind11/modules/tools/clang/config	remote.origin.url
2023-01-11T21:17:20.3893393Z Entering 'third_party/zstd'
2023-01-11T21:17:20.3934208Z file:/home/ec2-user/actions-runner/_work/pytorch/pytorch/.git/modules/third_party/zstd/config	remote.origin.url
2023-01-11T21:17:20.4696241Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'git@github.com:'
2023-01-11T21:17:20.5024434Z Entering 'android/libs/fbjni'
2023-01-11T21:17:20.5068605Z Entering 'third_party/FP16'
2023-01-11T21:17:20.5112582Z Entering 'third_party/FXdiv'
2023-01-11T21:17:20.5156791Z Entering 'third_party/NNPACK'
2023-01-11T21:17:20.5201006Z Entering 'third_party/QNNPACK'
2023-01-11T21:17:20.5246225Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T21:17:20.5291609Z Entering 'third_party/XNNPACK'
2023-01-11T21:17:20.5349276Z Entering 'third_party/benchmark'
2023-01-11T21:17:20.5394283Z Entering 'third_party/cpuinfo'
2023-01-11T21:17:20.5439701Z Entering 'third_party/cub'
2023-01-11T21:17:20.5484710Z Entering 'third_party/cudnn_frontend'
2023-01-11T21:17:20.5534663Z Entering 'third_party/cutlass'
2023-01-11T21:17:20.5585283Z Entering 'third_party/eigen'
2023-01-11T21:17:20.5632318Z Entering 'third_party/fbgemm'
2023-01-11T21:17:20.5676913Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:17:20.5719783Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:17:20.5763191Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:17:20.5806391Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:17:20.5853523Z Entering 'third_party/flatbuffers'
2023-01-11T21:17:20.5898807Z Entering 'third_party/fmt'
2023-01-11T21:17:20.5941822Z Entering 'third_party/foxi'
2023-01-11T21:17:20.5985510Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:17:20.6044646Z Entering 'third_party/gloo'
2023-01-11T21:17:20.6087717Z Entering 'third_party/googletest'
2023-01-11T21:17:20.6132366Z Entering 'third_party/ideep'
2023-01-11T21:17:20.6174543Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T21:17:20.6220669Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:17:20.6271359Z Entering 'third_party/ios-cmake'
2023-01-11T21:17:20.6315073Z Entering 'third_party/ittapi'
2023-01-11T21:17:20.6359678Z Entering 'third_party/kineto'
2023-01-11T21:17:20.6402967Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:17:20.6451163Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:17:20.6495434Z Entering 'third_party/nccl/nccl'
2023-01-11T21:17:20.6538493Z Entering 'third_party/neon2sse'
2023-01-11T21:17:20.6581163Z Entering 'third_party/nlohmann'
2023-01-11T21:17:20.6627288Z Entering 'third_party/onnx'
2023-01-11T21:17:20.6832871Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T21:17:20.6876759Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T21:17:20.6922284Z Entering 'third_party/onnx-tensorrt'
2023-01-11T21:17:20.6965874Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:17:20.7016050Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:17:20.7059511Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:17:20.7102322Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:17:20.7151727Z Entering 'third_party/pocketfft'
2023-01-11T21:17:20.7195476Z Entering 'third_party/protobuf'
2023-01-11T21:17:20.7261952Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:17:20.7306689Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T21:17:20.7352898Z Entering 'third_party/psimd'
2023-01-11T21:17:20.7396924Z Entering 'third_party/pthreadpool'
2023-01-11T21:17:20.7441557Z Entering 'third_party/pybind11'
2023-01-11T21:17:20.7485029Z Entering 'third_party/python-enum'
2023-01-11T21:17:20.7528825Z Entering 'third_party/python-peachpy'
2023-01-11T21:17:20.7573638Z Entering 'third_party/python-six'
2023-01-11T21:17:20.7618760Z Entering 'third_party/sleef'
2023-01-11T21:17:20.7662446Z Entering 'third_party/tbb'
2023-01-11T21:17:20.7718169Z Entering 'third_party/tensorpipe'
2023-01-11T21:17:20.7762158Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:17:20.7806894Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:17:20.7850717Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:17:20.7893883Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:17:20.7936881Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:17:20.7982737Z Entering 'third_party/zstd'
2023-01-11T21:17:20.8039589Z [command]/usr/bin/git submodule foreach --recursive git config --local --add 'url.https://github.com/.insteadOf' 'org-21003710@github.com:'
2023-01-11T21:17:20.8366297Z Entering 'android/libs/fbjni'
2023-01-11T21:17:20.8409706Z Entering 'third_party/FP16'
2023-01-11T21:17:20.8453862Z Entering 'third_party/FXdiv'
2023-01-11T21:17:20.8497739Z Entering 'third_party/NNPACK'
2023-01-11T21:17:20.8542712Z Entering 'third_party/QNNPACK'
2023-01-11T21:17:20.8586877Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T21:17:20.8637524Z Entering 'third_party/XNNPACK'
2023-01-11T21:17:20.8692443Z Entering 'third_party/benchmark'
2023-01-11T21:17:20.8737420Z Entering 'third_party/cpuinfo'
2023-01-11T21:17:20.8782000Z Entering 'third_party/cub'
2023-01-11T21:17:20.8826100Z Entering 'third_party/cudnn_frontend'
2023-01-11T21:17:20.8897549Z Entering 'third_party/cutlass'
2023-01-11T21:17:20.8948193Z Entering 'third_party/eigen'
2023-01-11T21:17:20.9022176Z Entering 'third_party/fbgemm'
2023-01-11T21:17:20.9066683Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T21:17:20.9108980Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T21:17:20.9153049Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T21:17:20.9195811Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T21:17:20.9241242Z Entering 'third_party/flatbuffers'
2023-01-11T21:17:20.9287702Z Entering 'third_party/fmt'
2023-01-11T21:17:20.9346544Z Entering 'third_party/foxi'
2023-01-11T21:17:20.9391154Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T21:17:20.9437215Z Entering 'third_party/gloo'
2023-01-11T21:17:20.9480700Z Entering 'third_party/googletest'
2023-01-11T21:17:20.9524865Z Entering 'third_party/ideep'
2023-01-11T21:17:20.9567345Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T21:17:20.9612484Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T21:17:20.9662846Z Entering 'third_party/ios-cmake'
2023-01-11T21:17:20.9708282Z Entering 'third_party/ittapi'
2023-01-11T21:17:20.9752282Z Entering 'third_party/kineto'
2023-01-11T21:17:20.9795272Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T21:17:20.9837917Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T21:17:20.9883039Z Entering 'third_party/nccl/nccl'
2023-01-11T21:17:20.9926127Z Entering 'third_party/neon2sse'
2023-01-11T21:17:20.9970399Z Entering 'third_party/nlohmann'
2023-01-11T21:17:21.0014726Z Entering 'third_party/onnx'
2023-01-11T21:17:21.0071730Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T21:17:21.0114957Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T21:17:21.0161931Z Entering 'third_party/onnx-tensorrt'
2023-01-11T21:17:21.0205474Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T21:17:21.0255618Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T21:17:21.0299188Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T21:17:21.0346741Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T21:17:21.0398648Z Entering 'third_party/pocketfft'
2023-01-11T21:17:21.0445507Z Entering 'third_party/protobuf'
2023-01-11T21:17:21.0495951Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T21:17:21.0539081Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T21:17:21.0584614Z Entering 'third_party/psimd'
2023-01-11T21:17:21.0629111Z Entering 'third_party/pthreadpool'
2023-01-11T21:17:21.0674433Z Entering 'third_party/pybind11'
2023-01-11T21:17:21.0719255Z Entering 'third_party/python-enum'
2023-01-11T21:17:21.0762992Z Entering 'third_party/python-peachpy'
2023-01-11T21:17:21.0808196Z Entering 'third_party/python-six'
2023-01-11T21:17:21.0852956Z Entering 'third_party/sleef'
2023-01-11T21:17:21.0897033Z Entering 'third_party/tbb'
2023-01-11T21:17:21.0942456Z Entering 'third_party/tensorpipe'
2023-01-11T21:17:21.0987175Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T21:17:21.1030455Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T21:17:21.1075136Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T21:17:21.1120852Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T21:17:21.1162461Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T21:17:21.1209341Z Entering 'third_party/zstd'
2023-01-11T21:17:21.1263907Z ##[endgroup]
2023-01-11T21:17:21.1310587Z [command]/usr/bin/git log -1 --format='%H'
2023-01-11T21:17:21.1339748Z '8419ddda87c8a47eacc63b54bc7ec98c1f27c26e'
2023-01-11T21:17:21.1486116Z Prepare all required actions
2023-01-11T21:17:21.1553642Z ##[group]Run ./.github/actions/setup-linux
2023-01-11T21:17:21.1553901Z env:
2023-01-11T21:17:21.1554137Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:17:21.1554393Z ##[endgroup]
2023-01-11T21:17:21.1584273Z ##[group]Run set -euo pipefail
2023-01-11T21:17:21.1584576Z set -euo pipefail
2023-01-11T21:17:21.1584838Z function get_ec2_metadata() {
2023-01-11T21:17:21.1585155Z   # Pulled from instance metadata endpoint for EC2
2023-01-11T21:17:21.1585613Z   # see https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/instancedata-data-retrieval.html
2023-01-11T21:17:21.1585988Z   category=$1
2023-01-11T21:17:21.1586317Z   curl -fsSL "http://169.254.169.254/latest/meta-data/${category}"
2023-01-11T21:17:21.1586611Z }
2023-01-11T21:17:21.1586863Z echo "ami-id: $(get_ec2_metadata ami-id)"
2023-01-11T21:17:21.1587236Z echo "instance-id: $(get_ec2_metadata instance-id)"
2023-01-11T21:17:21.1587614Z echo "instance-type: $(get_ec2_metadata instance-type)"
2023-01-11T21:17:21.1587930Z echo "system info $(uname -a)"
2023-01-11T21:17:21.1601026Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:17:21.1601304Z env:
2023-01-11T21:17:21.1601520Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:17:21.1601775Z ##[endgroup]
2023-01-11T21:17:21.1704879Z ami-id: ami-096198a0bccc6bad4
2023-01-11T21:17:21.1769807Z instance-id: i-0f914c3983ac93cd3
2023-01-11T21:17:21.1831501Z instance-type: g3.8xlarge
2023-01-11T21:17:21.1840767Z system info Linux ip-10-0-4-67.ec2.internal 4.14.252-195.483.amzn2.x86_64 #1 SMP Mon Nov 1 20:58:46 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux
2023-01-11T21:17:21.1859422Z ##[group]Run if systemctl is-active --quiet docker; then
2023-01-11T21:17:21.1859788Z if systemctl is-active --quiet docker; then
2023-01-11T21:17:21.1860122Z     echo "Docker daemon is running...";
2023-01-11T21:17:21.1860409Z else
2023-01-11T21:17:21.1860712Z     echo "Starting docker deamon..." && sudo systemctl start docker;
2023-01-11T21:17:21.1861020Z fi
2023-01-11T21:17:21.1873065Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:17:21.1873341Z env:
2023-01-11T21:17:21.1873579Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:17:21.1873839Z ##[endgroup]
2023-01-11T21:17:21.1925047Z Docker daemon is running...
2023-01-11T21:17:21.1942954Z ##[group]Run AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2023-01-11T21:17:21.1943428Z AWS_ACCOUNT_ID=$(aws sts get-caller-identity|grep Account|cut -f4 -d\")
2023-01-11T21:17:21.1943796Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-01-11T21:17:21.1944282Z retry aws ecr get-login*** "$AWS_DEFAULT_REGION" | docker login --username AWS \
2023-01-11T21:17:21.1944755Z     --password-stdin "$AWS_ACCOUNT_ID.dkr.ecr.$AWS_DEFAULT_REGION.amazonaws.com"
2023-01-11T21:17:21.1956447Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:17:21.1956717Z env:
2023-01-11T21:17:21.1956955Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:17:21.1957223Z   AWS_RETRY_MODE: standard
2023-01-11T21:17:21.1957454Z   AWS_MAX_ATTEMPTS: 5
2023-01-11T21:17:21.1957701Z   AWS_DEFAULT_REGION: us-east-1
2023-01-11T21:17:21.1957946Z ##[endgroup]
2023-01-11T21:17:22.1433098Z WARNING! Your password will be stored unencrypted in /home/ec2-user/.docker/config.json.
2023-01-11T21:17:22.1433587Z Configure a credential helper to remove this warning. See
2023-01-11T21:17:22.1434103Z https://docs.docker.com/engine/reference/commandline/login/#credentials-store
2023-01-11T21:17:22.1434376Z 
2023-01-11T21:17:22.1434870Z Login Succeeded
2023-01-11T21:17:22.1470162Z ##[group]Run env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-01-11T21:17:22.1470580Z env | grep '^GITHUB' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-01-11T21:17:22.1471093Z env | grep '^CI' >> "/tmp/github_env_${GITHUB_RUN_ID}"
2023-01-11T21:17:22.1485184Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:17:22.1485476Z env:
2023-01-11T21:17:22.1485721Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:17:22.1485983Z ##[endgroup]
2023-01-11T21:17:22.1576150Z ##[group]Run pytorch/test-infra/.github/actions/pull-docker-image@main
2023-01-11T21:17:22.1576518Z with:
2023-01-11T21:17:22.1577012Z   docker-image: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:17:22.1577471Z env:
2023-01-11T21:17:22.1577712Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:17:22.1577969Z ##[endgroup]
2023-01-11T21:17:22.1593171Z ##[group]Run retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-01-11T21:17:22.1593525Z retry () { "$@"  || (sleep 1 && "$@") || (sleep 2 && "$@") }
2023-01-11T21:17:22.1593888Z # ignore output since only exit code is used for conditional
2023-01-11T21:17:22.1594272Z # only pull docker image if it's not available locally
2023-01-11T21:17:22.1594673Z if ! docker inspect --type=image "${DOCKER_IMAGE}" >/dev/null 2>/dev/null; then
2023-01-11T21:17:22.1595070Z   retry docker pull "${DOCKER_IMAGE}"
2023-01-11T21:17:22.1595342Z fi
2023-01-11T21:17:22.1608071Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:17:22.1608369Z env:
2023-01-11T21:17:22.1608612Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:17:22.1609124Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:17:22.1609616Z ##[endgroup]
2023-01-11T21:17:22.4121358Z fd224c2e6c79d7fdec6408da598bf52bc5b201dd: Pulling from pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7
2023-01-11T21:17:22.4121862Z fb668870d8a7: Pulling fs layer
2023-01-11T21:17:22.4122196Z 4542784317be: Pulling fs layer
2023-01-11T21:17:22.4122706Z e0bec5df5af5: Pulling fs layer
2023-01-11T21:17:22.4123193Z 4053f75740ab: Pulling fs layer
2023-01-11T21:17:22.4123472Z 57e09105cdfd: Pulling fs layer
2023-01-11T21:17:22.4123723Z 606761d225e5: Pulling fs layer
2023-01-11T21:17:22.4124003Z 69473a703fb4: Pulling fs layer
2023-01-11T21:17:22.4124599Z a08ab4e0594b: Pulling fs layer
2023-01-11T21:17:22.4124870Z 4cd507bccac2: Pulling fs layer
2023-01-11T21:17:22.4130077Z fa92f16621a4: Pulling fs layer
2023-01-11T21:17:22.4130750Z 6dc2b05bd224: Pulling fs layer
2023-01-11T21:17:22.4131312Z ce4a87d45645: Pulling fs layer
2023-01-11T21:17:22.4131821Z 41860ea59b6c: Pulling fs layer
2023-01-11T21:17:22.4132188Z 87d0ffa55850: Pulling fs layer
2023-01-11T21:17:22.4132482Z f9f75aaba8d7: Pulling fs layer
2023-01-11T21:17:22.4132978Z 0c06be5c20e0: Pulling fs layer
2023-01-11T21:17:22.4133389Z d23c0a07b67c: Pulling fs layer
2023-01-11T21:17:22.4133824Z 1001f0d2f3d0: Pulling fs layer
2023-01-11T21:17:22.4134246Z e1c655e7ec0e: Pulling fs layer
2023-01-11T21:17:22.4134883Z a11b4b5fd784: Pulling fs layer
2023-01-11T21:17:22.4135342Z bc41eab7f454: Pulling fs layer
2023-01-11T21:17:22.4135784Z b8f759fd0191: Pulling fs layer
2023-01-11T21:17:22.4136209Z f410dcc9d0be: Pulling fs layer
2023-01-11T21:17:22.4149728Z 57e09105cdfd: Waiting
2023-01-11T21:17:22.4150399Z 90d8f9bbe048: Pulling fs layer
2023-01-11T21:17:22.4150715Z eedfbaa04e4f: Pulling fs layer
2023-01-11T21:17:22.4150997Z 2f2308643d60: Pulling fs layer
2023-01-11T21:17:22.4151268Z c1a92fad2c2c: Pulling fs layer
2023-01-11T21:17:22.4151526Z 606761d225e5: Waiting
2023-01-11T21:17:22.4151785Z 47037a50f270: Pulling fs layer
2023-01-11T21:17:22.4152042Z 1a2fd7b216d7: Pulling fs layer
2023-01-11T21:17:22.4152303Z 4cd507bccac2: Waiting
2023-01-11T21:17:22.4152561Z 765839304d2e: Pulling fs layer
2023-01-11T21:17:22.4152814Z e51794baeb92: Pulling fs layer
2023-01-11T21:17:22.4153077Z fa92f16621a4: Waiting
2023-01-11T21:17:22.4153320Z 69473a703fb4: Waiting
2023-01-11T21:17:22.4153553Z 87d0ffa55850: Waiting
2023-01-11T21:17:22.4153795Z b8f759fd0191: Waiting
2023-01-11T21:17:22.4154037Z 0c06be5c20e0: Waiting
2023-01-11T21:17:22.4154258Z 6dc2b05bd224: Waiting
2023-01-11T21:17:22.4154497Z 90d8f9bbe048: Waiting
2023-01-11T21:17:22.4154738Z f410dcc9d0be: Waiting
2023-01-11T21:17:22.4155127Z 4053f75740ab: Waiting
2023-01-11T21:17:22.4155407Z ea4bfeaa0fc7: Pulling fs layer
2023-01-11T21:17:22.4155672Z ce4a87d45645: Waiting
2023-01-11T21:17:22.4161813Z a08ab4e0594b: Waiting
2023-01-11T21:17:22.4162174Z eedfbaa04e4f: Waiting
2023-01-11T21:17:22.4162439Z d23c0a07b67c: Waiting
2023-01-11T21:17:22.4162693Z 1001f0d2f3d0: Waiting
2023-01-11T21:17:22.4162922Z a11b4b5fd784: Waiting
2023-01-11T21:17:22.4163167Z 2f2308643d60: Waiting
2023-01-11T21:17:22.4163406Z 1a2fd7b216d7: Waiting
2023-01-11T21:17:22.4163632Z bc41eab7f454: Waiting
2023-01-11T21:17:22.4163876Z c1a92fad2c2c: Waiting
2023-01-11T21:17:22.4164117Z 47037a50f270: Waiting
2023-01-11T21:17:22.4164759Z e1c655e7ec0e: Waiting
2023-01-11T21:17:22.4165270Z d8065d17513d: Pulling fs layer
2023-01-11T21:17:22.4165794Z 6d83ca3dedf3: Pulling fs layer
2023-01-11T21:17:22.4166316Z 12ddc57b99eb: Pulling fs layer
2023-01-11T21:17:22.4166806Z b590670d273c: Pulling fs layer
2023-01-11T21:17:22.4167291Z ea4bfeaa0fc7: Waiting
2023-01-11T21:17:22.4167573Z 8afbc57dfec9: Pulling fs layer
2023-01-11T21:17:22.4167837Z 41860ea59b6c: Waiting
2023-01-11T21:17:22.4168075Z d8065d17513d: Waiting
2023-01-11T21:17:22.4168296Z 12ddc57b99eb: Waiting
2023-01-11T21:17:22.4168537Z b590670d273c: Waiting
2023-01-11T21:17:22.4168790Z 29a7c0d5fa4c: Pulling fs layer
2023-01-11T21:17:22.4169043Z 16825bb02017: Pulling fs layer
2023-01-11T21:17:22.4169323Z bdf297d7f88c: Pulling fs layer
2023-01-11T21:17:22.4169594Z 885c12efa4ae: Pulling fs layer
2023-01-11T21:17:22.4169843Z 28c5689cb975: Pulling fs layer
2023-01-11T21:17:22.4170118Z cca768f96df4: Pulling fs layer
2023-01-11T21:17:22.4170391Z 904b81494b5e: Pulling fs layer
2023-01-11T21:17:22.4170662Z 61eecfa8b34e: Pulling fs layer
2023-01-11T21:17:22.4170922Z 95c1ac011645: Pulling fs layer
2023-01-11T21:17:22.4171175Z 29a7c0d5fa4c: Waiting
2023-01-11T21:17:22.4171439Z 07cee023724c: Pulling fs layer
2023-01-11T21:17:22.4171685Z 195d560d8cf6: Pulling fs layer
2023-01-11T21:17:22.4171949Z a399389c7f8e: Pulling fs layer
2023-01-11T21:17:22.4172211Z 885c12efa4ae: Waiting
2023-01-11T21:17:22.4172449Z 7447f84b33ef: Pulling fs layer
2023-01-11T21:17:22.4172704Z 07cee023724c: Waiting
2023-01-11T21:17:22.4172949Z 8afbc57dfec9: Waiting
2023-01-11T21:17:22.4173174Z bdf297d7f88c: Waiting
2023-01-11T21:17:22.4173420Z 61eecfa8b34e: Waiting
2023-01-11T21:17:22.4173685Z 0d8aeb1421f9: Pulling fs layer
2023-01-11T21:17:22.4173924Z cca768f96df4: Waiting
2023-01-11T21:17:22.4174182Z 95c1ac011645: Waiting
2023-01-11T21:17:22.4174434Z 02048a597c22: Pulling fs layer
2023-01-11T21:17:22.4174666Z 16825bb02017: Waiting
2023-01-11T21:17:22.4174917Z 25d615d8a5e2: Pulling fs layer
2023-01-11T21:17:22.4175170Z a399389c7f8e: Waiting
2023-01-11T21:17:22.4175409Z 09d400b86049: Pulling fs layer
2023-01-11T21:17:22.4175847Z 25d615d8a5e2: Waiting
2023-01-11T21:17:22.4176097Z 195d560d8cf6: Waiting
2023-01-11T21:17:22.4176318Z 0d8aeb1421f9: Waiting
2023-01-11T21:17:22.4176558Z 904b81494b5e: Waiting
2023-01-11T21:17:22.5756239Z 4542784317be: Download complete
2023-01-11T21:17:22.6739572Z 4053f75740ab: Download complete
2023-01-11T21:17:22.7322116Z fb668870d8a7: Verifying Checksum
2023-01-11T21:17:22.7323177Z fb668870d8a7: Download complete
2023-01-11T21:17:22.7536178Z 57e09105cdfd: Download complete
2023-01-11T21:17:22.8647697Z 69473a703fb4: Verifying Checksum
2023-01-11T21:17:22.8648204Z 69473a703fb4: Download complete
2023-01-11T21:17:22.9765331Z e0bec5df5af5: Verifying Checksum
2023-01-11T21:17:22.9765843Z e0bec5df5af5: Download complete
2023-01-11T21:17:22.9955664Z a08ab4e0594b: Download complete
2023-01-11T21:17:23.0855994Z 4cd507bccac2: Verifying Checksum
2023-01-11T21:17:23.0856350Z 4cd507bccac2: Download complete
2023-01-11T21:17:23.1651665Z 6dc2b05bd224: Verifying Checksum
2023-01-11T21:17:23.1652323Z 6dc2b05bd224: Download complete
2023-01-11T21:17:23.2488529Z ce4a87d45645: Verifying Checksum
2023-01-11T21:17:23.2488904Z ce4a87d45645: Download complete
2023-01-11T21:17:23.6813181Z fb668870d8a7: Pull complete
2023-01-11T21:17:23.9595260Z 4542784317be: Pull complete
2023-01-11T21:17:24.8578972Z e0bec5df5af5: Pull complete
2023-01-11T21:17:24.9800376Z 4053f75740ab: Pull complete
2023-01-11T21:17:25.1092671Z 57e09105cdfd: Pull complete
2023-01-11T21:17:25.2803346Z 41860ea59b6c: Verifying Checksum
2023-01-11T21:17:25.2803677Z 41860ea59b6c: Download complete
2023-01-11T21:17:25.3667001Z 87d0ffa55850: Verifying Checksum
2023-01-11T21:17:25.3667332Z 87d0ffa55850: Download complete
2023-01-11T21:17:25.4750373Z f9f75aaba8d7: Verifying Checksum
2023-01-11T21:17:25.4750723Z f9f75aaba8d7: Download complete
2023-01-11T21:17:25.5556905Z 0c06be5c20e0: Verifying Checksum
2023-01-11T21:17:25.5557220Z 0c06be5c20e0: Download complete
2023-01-11T21:17:26.3344704Z d23c0a07b67c: Verifying Checksum
2023-01-11T21:17:26.3345086Z d23c0a07b67c: Download complete
2023-01-11T21:17:26.4194421Z 1001f0d2f3d0: Download complete
2023-01-11T21:17:26.4940819Z e1c655e7ec0e: Verifying Checksum
2023-01-11T21:17:26.4941179Z e1c655e7ec0e: Download complete
2023-01-11T21:17:33.6456807Z 606761d225e5: Verifying Checksum
2023-01-11T21:17:33.6457171Z 606761d225e5: Download complete
2023-01-11T21:17:33.7633889Z bc41eab7f454: Verifying Checksum
2023-01-11T21:17:33.7634332Z bc41eab7f454: Download complete
2023-01-11T21:17:33.8447184Z b8f759fd0191: Verifying Checksum
2023-01-11T21:17:33.8447502Z b8f759fd0191: Download complete
2023-01-11T21:17:33.9321516Z f410dcc9d0be: Verifying Checksum
2023-01-11T21:17:33.9321897Z f410dcc9d0be: Download complete
2023-01-11T21:17:34.0173055Z 90d8f9bbe048: Verifying Checksum
2023-01-11T21:17:34.0173655Z 90d8f9bbe048: Download complete
2023-01-11T21:17:34.0978409Z eedfbaa04e4f: Verifying Checksum
2023-01-11T21:17:34.0979065Z eedfbaa04e4f: Download complete
2023-01-11T21:17:34.1785169Z 2f2308643d60: Download complete
2023-01-11T21:17:35.1235575Z c1a92fad2c2c: Verifying Checksum
2023-01-11T21:17:35.1235941Z c1a92fad2c2c: Download complete
2023-01-11T21:17:35.2301977Z 47037a50f270: Download complete
2023-01-11T21:17:35.3051292Z 1a2fd7b216d7: Verifying Checksum
2023-01-11T21:17:35.3051935Z 1a2fd7b216d7: Download complete
2023-01-11T21:17:35.3868447Z 765839304d2e: Verifying Checksum
2023-01-11T21:17:35.3868794Z 765839304d2e: Download complete
2023-01-11T21:17:35.4642450Z e51794baeb92: Verifying Checksum
2023-01-11T21:17:35.4642818Z e51794baeb92: Download complete
2023-01-11T21:17:35.5317417Z ea4bfeaa0fc7: Verifying Checksum
2023-01-11T21:17:35.5317768Z ea4bfeaa0fc7: Download complete
2023-01-11T21:17:37.1679176Z fa92f16621a4: Verifying Checksum
2023-01-11T21:17:37.2532041Z 6d83ca3dedf3: Download complete
2023-01-11T21:17:37.3237675Z 12ddc57b99eb: Verifying Checksum
2023-01-11T21:17:37.3238036Z 12ddc57b99eb: Download complete
2023-01-11T21:17:37.5270187Z d8065d17513d: Verifying Checksum
2023-01-11T21:17:37.5271063Z d8065d17513d: Download complete
2023-01-11T21:17:37.6122794Z 8afbc57dfec9: Verifying Checksum
2023-01-11T21:17:37.6123161Z 8afbc57dfec9: Download complete
2023-01-11T21:17:37.7195950Z 29a7c0d5fa4c: Verifying Checksum
2023-01-11T21:17:37.7196307Z 29a7c0d5fa4c: Download complete
2023-01-11T21:17:37.7236811Z b590670d273c: Verifying Checksum
2023-01-11T21:17:37.7237505Z b590670d273c: Download complete
2023-01-11T21:17:37.8096526Z bdf297d7f88c: Verifying Checksum
2023-01-11T21:17:37.8096863Z bdf297d7f88c: Download complete
2023-01-11T21:17:37.9810562Z 16825bb02017: Verifying Checksum
2023-01-11T21:17:37.9810918Z 16825bb02017: Download complete
2023-01-11T21:17:38.0721770Z 28c5689cb975: Verifying Checksum
2023-01-11T21:17:38.0722275Z 28c5689cb975: Download complete
2023-01-11T21:17:38.1556829Z cca768f96df4: Download complete
2023-01-11T21:17:38.3080393Z 885c12efa4ae: Verifying Checksum
2023-01-11T21:17:38.3081139Z 885c12efa4ae: Download complete
2023-01-11T21:17:38.3879217Z 61eecfa8b34e: Verifying Checksum
2023-01-11T21:17:38.3879568Z 61eecfa8b34e: Download complete
2023-01-11T21:17:38.4695658Z 95c1ac011645: Download complete
2023-01-11T21:17:38.5464468Z 07cee023724c: Download complete
2023-01-11T21:17:38.6288478Z 195d560d8cf6: Download complete
2023-01-11T21:17:38.8181153Z a399389c7f8e: Verifying Checksum
2023-01-11T21:17:38.8181996Z a399389c7f8e: Download complete
2023-01-11T21:17:38.9009421Z 7447f84b33ef: Verifying Checksum
2023-01-11T21:17:39.4982583Z 0d8aeb1421f9: Verifying Checksum
2023-01-11T21:17:39.4982954Z 0d8aeb1421f9: Download complete
2023-01-11T21:17:39.5714561Z 02048a597c22: Download complete
2023-01-11T21:17:40.9258763Z 904b81494b5e: Verifying Checksum
2023-01-11T21:17:40.9259368Z 904b81494b5e: Download complete
2023-01-11T21:17:41.0038055Z 09d400b86049: Verifying Checksum
2023-01-11T21:17:41.0038418Z 09d400b86049: Download complete
2023-01-11T21:17:47.6053190Z 606761d225e5: Pull complete
2023-01-11T21:17:47.7206383Z 69473a703fb4: Pull complete
2023-01-11T21:17:47.8365150Z a08ab4e0594b: Pull complete
2023-01-11T21:17:47.9598704Z 4cd507bccac2: Pull complete
2023-01-11T21:18:10.0713526Z fa92f16621a4: Pull complete
2023-01-11T21:18:11.9505869Z 6dc2b05bd224: Pull complete
2023-01-11T21:18:14.0402067Z ce4a87d45645: Pull complete
2023-01-11T21:18:22.0361492Z 41860ea59b6c: Pull complete
2023-01-11T21:18:24.0724549Z 87d0ffa55850: Pull complete
2023-01-11T21:18:25.2047237Z a11b4b5fd784: Verifying Checksum
2023-01-11T21:18:26.0750848Z f9f75aaba8d7: Pull complete
2023-01-11T21:18:28.3217796Z 0c06be5c20e0: Pull complete
2023-01-11T21:18:32.5957074Z d23c0a07b67c: Pull complete
2023-01-11T21:18:34.9778202Z 1001f0d2f3d0: Pull complete
2023-01-11T21:18:36.8878700Z e1c655e7ec0e: Pull complete
2023-01-11T21:18:56.6641650Z 25d615d8a5e2: Verifying Checksum
2023-01-11T21:18:56.6644894Z 25d615d8a5e2: Download complete
2023-01-11T21:19:10.4371387Z a11b4b5fd784: Pull complete
2023-01-11T21:19:12.3121443Z bc41eab7f454: Pull complete
2023-01-11T21:19:14.0737219Z b8f759fd0191: Pull complete
2023-01-11T21:19:15.9015109Z f410dcc9d0be: Pull complete
2023-01-11T21:19:17.7341162Z 90d8f9bbe048: Pull complete
2023-01-11T21:19:20.3376897Z eedfbaa04e4f: Pull complete
2023-01-11T21:19:22.2026867Z 2f2308643d60: Pull complete
2023-01-11T21:19:26.4244310Z c1a92fad2c2c: Pull complete
2023-01-11T21:19:28.2984769Z 47037a50f270: Pull complete
2023-01-11T21:19:30.5297198Z 1a2fd7b216d7: Pull complete
2023-01-11T21:19:33.3491924Z 765839304d2e: Pull complete
2023-01-11T21:19:36.7147005Z e51794baeb92: Pull complete
2023-01-11T21:19:39.5279839Z ea4bfeaa0fc7: Pull complete
2023-01-11T21:19:48.3112413Z d8065d17513d: Pull complete
2023-01-11T21:19:50.1898145Z 6d83ca3dedf3: Pull complete
2023-01-11T21:19:52.0977338Z 12ddc57b99eb: Pull complete
2023-01-11T21:19:54.9527047Z b590670d273c: Pull complete
2023-01-11T21:19:55.8435348Z 8afbc57dfec9: Pull complete
2023-01-11T21:19:55.9759421Z 29a7c0d5fa4c: Pull complete
2023-01-11T21:19:56.3433890Z 16825bb02017: Pull complete
2023-01-11T21:19:56.4324730Z bdf297d7f88c: Pull complete
2023-01-11T21:19:57.8712293Z 885c12efa4ae: Pull complete
2023-01-11T21:19:57.9765612Z 28c5689cb975: Pull complete
2023-01-11T21:19:58.0801742Z cca768f96df4: Pull complete
2023-01-11T21:20:02.9156538Z 904b81494b5e: Pull complete
2023-01-11T21:20:03.0358924Z 61eecfa8b34e: Pull complete
2023-01-11T21:20:03.1368570Z 95c1ac011645: Pull complete
2023-01-11T21:20:03.2366959Z 07cee023724c: Pull complete
2023-01-11T21:20:03.3489246Z 195d560d8cf6: Pull complete
2023-01-11T21:20:04.1386163Z a399389c7f8e: Pull complete
2023-01-11T21:20:04.2360513Z 7447f84b33ef: Pull complete
2023-01-11T21:20:06.1586031Z 0d8aeb1421f9: Pull complete
2023-01-11T21:20:06.2545301Z 02048a597c22: Pull complete
2023-01-11T21:20:36.5846832Z 25d615d8a5e2: Pull complete
2023-01-11T21:20:38.4567225Z 09d400b86049: Pull complete
2023-01-11T21:20:39.7092338Z Digest: sha256:0da23f4faf0ce20770149c4a783e08eaa91c07112511dc5511c77937c66edb24
2023-01-11T21:20:40.2094362Z Status: Downloaded newer image for 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:20:40.4914620Z 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:20:40.5020562Z ##[group]Run pytorch/test-infra/.github/actions/setup-nvidia@main
2023-01-11T21:20:40.5020901Z with:
2023-01-11T21:20:40.5021146Z   driver-version: 515.76
2023-01-11T21:20:40.5021389Z env:
2023-01-11T21:20:40.5021609Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:20:40.5021873Z ##[endgroup]
2023-01-11T21:20:40.6492591Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2023-01-11T21:20:40.6492906Z with:
2023-01-11T21:20:40.6493137Z   timeout_minutes: 10
2023-01-11T21:20:40.6493387Z   max_attempts: 3
2023-01-11T21:20:40.6500365Z   command: # Is it disgusting to have a full shell script here in this github action? Sure
# But is it the best way to make it so that this action relies on nothing else? Absolutely
set -eou pipefail

DISTRIBUTION=$(. /etc/os-release;echo $ID$VERSION_ID)
DRIVER_FN="NVIDIA-Linux-x86_64-${DRIVER_VERSION}.run"
YUM_REPO_URL="https://nvidia.github.io/nvidia-docker/${DISTRIBUTION}/nvidia-docker.repo"

install_nvidia_docker2_amzn2() {
    (
        set -x
        # Needed for yum-config-manager
        sudo yum install -y yum-utils
        sudo yum-config-manager --add-repo "${YUM_REPO_URL}"
        sudo yum install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

install_nvidia_docker2_ubuntu20() {
    (
        set -x
        sudo apt-get install -y nvidia-docker2
        sudo systemctl restart docker
    )
}

pre_install_nvidia_driver_amzn2() {
    (
        # Purge any nvidia driver installed from RHEL repo
        sudo yum remove -y nvidia-driver-latest-dkms
    )
}

install_nvidia_driver_common() {
    (
        # Try to gather more information about the runner and its existing NVIDIA driver if any
        echo "Before installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        HAS_NVIDIA_DRIVER=0
        # Check if NVIDIA driver has already been installed
        if [ -x "$(command -v nvidia-smi)" ]; then
            set +e
            # The driver exists, check its version next. Also check only the first GPU if there are more than one of them
            # so that the same driver version is not print over multiple lines
            INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
            NVIDIA_SMI_STATUS=$?

            if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                echo "Failed to get NVIDIA driver version ($INSTALLED_DRIVER_VERSION). Continuing"
            elif [ "$INSTALLED_DRIVER_VERSION" != "$DRIVER_VERSION" ]; then
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has been installed, but we expect to have $DRIVER_VERSION instead. Continuing"
            else
                HAS_NVIDIA_DRIVER=1
                echo "NVIDIA driver ($INSTALLED_DRIVER_VERSION) has already been installed. Skipping NVIDIA driver installation"
            fi
            set -e
        fi

        if [ "$HAS_NVIDIA_DRIVER" -eq 0 ]; then
            # CAUTION: this may need to be updated in future
            if [ "${DISTRIBUTION}" != ubuntu20.04 ]; then
                  sudo yum groupinstall -y "Development Tools"
                  # ensure our kernel install is the same as our underlying kernel,
                  # groupinstall "Development Tools" has a habit of mismatching kernel headers
                  sudo yum install -y "kernel-devel-uname-r == $(uname -r)"
                  sudo modprobe backlight
            fi
            sudo curl -fsL -o /tmp/nvidia_driver "https://s3.amazonaws.com/ossci-linux/nvidia_driver/$DRIVER_FN"

            set +e
            sudo /bin/bash /tmp/nvidia_driver -s --no-drm
            NVIDIA_INSTALLATION_STATUS=$?

            RESET_GPU=0
            if [ "$NVIDIA_INSTALLATION_STATUS" -ne 0 ]; then
                sudo cat /var/log/nvidia-installer.log
                # Fail to install NVIDIA driver, try to reset the GPU
                RESET_GPU=1
            elif [ -x "$(command -v nvidia-smi)" ]; then
                # Check again if nvidia-smi works even if the driver installation completes successfully
                INSTALLED_DRIVER_VERSION=$(nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0)
                NVIDIA_SMI_STATUS=$?

                if [ "$NVIDIA_SMI_STATUS" -ne 0 ] && [ "$NVIDIA_SMI_STATUS" -ne 14 ]; then
                    RESET_GPU=1
                fi
            fi

            if [ "$RESET_GPU" -eq 1 ]; then
                NVIDIA_DEVICES=$(lspci -D | grep -i NVIDIA | cut -d' ' -f1)
                # The GPU can get stuck in a failure state if somehow the test crashs the GPU microcode. When this
                # happens, we'll try to reset all NVIDIA devices https://github.com/pytorch/pytorch/issues/88388
                for PCI_ID in $NVIDIA_DEVICES; do
                    DEVICE_ENABLED=$(cat /sys/bus/pci/devices/$PCI_ID/enable)

                    echo "Reseting $PCI_ID (enabled state: $DEVICE_ENABLED)"
                    # This requires sudo permission of course
                    echo "1" | sudo tee /sys/bus/pci/devices/$PCI_ID/reset
                    sleep 1
                done
            fi

            sudo rm -fv /tmp/nvidia_driver
            set -e
        fi
    )
}

post_install_nvidia_driver_common() {
    (
        sudo modprobe nvidia || true
        echo "After installing NVIDIA driver"
        lspci
        lsmod
        modinfo nvidia || true

        (
            set +e
            nvidia-smi
            NVIDIA_SMI_STATUS=$?

            # Allowable exit statuses for nvidia-smi, see: https://github.com/NVIDIA/gpu-operator/issues/285
            if [ "$NVIDIA_SMI_STATUS" -eq 0 ] || [ "$NVIDIA_SMI_STATUS" -eq 14 ]; then
                echo "INFO: Ignoring allowed status ${NVIDIA_SMI_STATUS}"
            else
                echo "ERROR: nvidia-smi exited with unresolved status ${NVIDIA_SMI_STATUS}"
                exit ${NVIDIA_SMI_STATUS}
            fi
            set -e
        )
    )
}

install_nvidia_driver_amzn2() {
    (
        set -x
        pre_install_nvidia_driver_amzn2
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

install_nvidia_driver_ubuntu20() {
    (
        set -x
        install_nvidia_driver_common
        post_install_nvidia_driver_common
    )
}

echo "== Installing nvidia driver ${DRIVER_FN} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_driver_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_driver_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac

# Install container toolkit based on distribution
echo "== Installing nvidia container toolkit for ${DISTRIBUTION} =="
case "${DISTRIBUTION}" in
    amzn*)
        install_nvidia_docker2_amzn2
        ;;
    ubuntu20.04)
        install_nvidia_docker2_ubuntu20
        ;;
    *)
        echo "ERROR: Unknown distribution ${DISTRIBUTION}"
        exit 1
        ;;
esac
echo "GPU_FLAG=--gpus all" >> "${GITHUB_ENV}"

2023-01-11T21:20:40.6507549Z   retry_wait_seconds: 10
2023-01-11T21:20:40.6507836Z   polling_interval_seconds: 1
2023-01-11T21:20:40.6508111Z   warning_on_retry: true
2023-01-11T21:20:40.6508357Z   continue_on_error: false
2023-01-11T21:20:40.6508598Z env:
2023-01-11T21:20:40.6508836Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:20:40.6509086Z   DRIVER_VERSION: 515.76
2023-01-11T21:20:40.6509343Z ##[endgroup]
2023-01-11T21:20:40.7232140Z == Installing nvidia driver NVIDIA-Linux-x86_64-515.76.run ==
2023-01-11T21:20:40.7234525Z + pre_install_nvidia_driver_amzn2
2023-01-11T21:20:40.7235232Z + sudo yum remove -y nvidia-driver-latest-dkms
2023-01-11T21:20:41.2877845Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2023-01-11T21:20:41.3453861Z No Match for argument: nvidia-driver-latest-dkms
2023-01-11T21:20:41.3831662Z No Packages marked for removal
2023-01-11T21:20:41.3992962Z + install_nvidia_driver_common
2023-01-11T21:20:41.3996250Z + echo 'Before installing NVIDIA driver'
2023-01-11T21:20:41.3996758Z Before installing NVIDIA driver
2023-01-11T21:20:41.3998945Z + lspci
2023-01-11T21:20:41.4191208Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2023-01-11T21:20:41.4191924Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2023-01-11T21:20:41.4192616Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
2023-01-11T21:20:41.4193224Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
2023-01-11T21:20:41.4193804Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
2023-01-11T21:20:41.4194402Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2023-01-11T21:20:41.4195066Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2023-01-11T21:20:41.4195736Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2023-01-11T21:20:41.4196444Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
2023-01-11T21:20:41.4196948Z + lsmod
2023-01-11T21:20:41.4214250Z Module                  Size  Used by
2023-01-11T21:20:41.4214746Z nvidia_modeset       1142784  0
2023-01-11T21:20:41.4215164Z nvidia_uvm           1269760  0
2023-01-11T21:20:41.4215566Z veth                   16384  0
2023-01-11T21:20:41.4216053Z nvidia              40808448  15 nvidia_uvm,nvidia_modeset
2023-01-11T21:20:41.4216499Z drm                   425984  1 nvidia
2023-01-11T21:20:41.4216948Z i2c_core               77824  2 nvidia,drm
2023-01-11T21:20:41.4217412Z backlight              16384  1 nvidia_modeset
2023-01-11T21:20:41.4217900Z xt_conntrack           16384  1
2023-01-11T21:20:41.4218311Z ipt_MASQUERADE         16384  1
2023-01-11T21:20:41.4218806Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2023-01-11T21:20:41.4219315Z nf_conntrack_netlink    49152  0
2023-01-11T21:20:41.4219829Z nfnetlink              16384  2 nf_conntrack_netlink
2023-01-11T21:20:41.4220338Z xfrm_user              45056  1
2023-01-11T21:20:41.4220783Z xfrm_algo              16384  1 xfrm_user
2023-01-11T21:20:41.4221181Z xt_addrtype            16384  2
2023-01-11T21:20:41.4221635Z iptable_filter         16384  1
2023-01-11T21:20:41.4222096Z iptable_nat            16384  1
2023-01-11T21:20:41.4222539Z nf_conntrack_ipv4      16384  3
2023-01-11T21:20:41.4223021Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2023-01-11T21:20:41.4223521Z nf_nat_ipv4            16384  1 iptable_nat
2023-01-11T21:20:41.4224022Z nf_nat                 36864  2 nf_nat_masquerade_ipv4,nf_nat_ipv4
2023-01-11T21:20:41.4224736Z nf_conntrack          155648  7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink
2023-01-11T21:20:41.4225386Z br_netfilter           24576  0
2023-01-11T21:20:41.4225847Z bridge                172032  1 br_netfilter
2023-01-11T21:20:41.4226283Z stp                    16384  1 bridge
2023-01-11T21:20:41.4226985Z llc                    16384  2 bridge,stp
2023-01-11T21:20:41.4227395Z overlay                86016  0
2023-01-11T21:20:41.4227743Z sunrpc                393216  1
2023-01-11T21:20:41.4228150Z dm_mirror              28672  0
2023-01-11T21:20:41.4228575Z dm_region_hash         20480  1 dm_mirror
2023-01-11T21:20:41.4229032Z dm_log                 20480  2 dm_region_hash,dm_mirror
2023-01-11T21:20:41.4229474Z dm_mod                143360  2 dm_log,dm_mirror
2023-01-11T21:20:41.4229903Z dax                    69632  1 dm_mod
2023-01-11T21:20:41.4230310Z sb_edac                24576  0
2023-01-11T21:20:41.4230644Z crc32_pclmul           16384  0
2023-01-11T21:20:41.4231016Z ghash_clmulni_intel    16384  0
2023-01-11T21:20:41.4231382Z pcbc                   16384  0
2023-01-11T21:20:41.4231715Z aesni_intel           188416  0
2023-01-11T21:20:41.4232087Z aes_x86_64             20480  1 aesni_intel
2023-01-11T21:20:41.4232452Z ata_piix               36864  0
2023-01-11T21:20:41.4232884Z crypto_simd            16384  1 aesni_intel
2023-01-11T21:20:41.4233276Z glue_helper            16384  1 aesni_intel
2023-01-11T21:20:41.4233664Z pcc_cpufreq            16384  0
2023-01-11T21:20:41.4234146Z libata                266240  1 ata_piix
2023-01-11T21:20:41.4234607Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2023-01-11T21:20:41.4235051Z mousedev               24576  0
2023-01-11T21:20:41.4235415Z scsi_mod              245760  1 libata
2023-01-11T21:20:41.4235763Z psmouse                32768  0
2023-01-11T21:20:41.4236113Z evdev                  20480  3
2023-01-11T21:20:41.4236455Z button                 16384  0
2023-01-11T21:20:41.4236782Z ena                   114688  0
2023-01-11T21:20:41.4237139Z xen_blkfront           49152  2
2023-01-11T21:20:41.4237497Z crc32c_intel           24576  0
2023-01-11T21:20:41.4237837Z autofs4                49152  2
2023-01-11T21:20:41.4238181Z + modinfo nvidia
2023-01-11T21:20:41.4238841Z filename:       /lib/modules/4.14.252-195.483.amzn2.x86_64/kernel/drivers/video/nvidia.ko
2023-01-11T21:20:41.4239329Z firmware:       nvidia/515.76/gsp.bin
2023-01-11T21:20:41.4239778Z alias:          char-major-195-*
2023-01-11T21:20:41.4240146Z version:        515.76
2023-01-11T21:20:41.4240499Z supported:      external
2023-01-11T21:20:41.4240831Z license:        NVIDIA
2023-01-11T21:20:41.4241204Z srcversion:     51FD9DD90150B35351AFFBB
2023-01-11T21:20:41.4241636Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2023-01-11T21:20:41.4242045Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2023-01-11T21:20:41.4242468Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2023-01-11T21:20:41.4242941Z depends:        i2c-core,drm
2023-01-11T21:20:41.4243281Z retpoline:      Y
2023-01-11T21:20:41.4243622Z name:           nvidia
2023-01-11T21:20:41.4244185Z vermagic:       4.14.252-195.483.amzn2.x86_64 SMP mod_unload modversions 
2023-01-11T21:20:41.4245257Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2023-01-11T21:20:41.4245814Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2023-01-11T21:20:41.4246384Z parm:           NVreg_ResmanDebugLevel:int
2023-01-11T21:20:41.4246796Z parm:           NVreg_RmLogonRC:int
2023-01-11T21:20:41.4247190Z parm:           NVreg_ModifyDeviceFiles:int
2023-01-11T21:20:41.4247616Z parm:           NVreg_DeviceFileUID:int
2023-01-11T21:20:41.4248034Z parm:           NVreg_DeviceFileGID:int
2023-01-11T21:20:41.4248429Z parm:           NVreg_DeviceFileMode:int
2023-01-11T21:20:41.4248928Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2023-01-11T21:20:41.4249450Z parm:           NVreg_UsePageAttributeTable:int
2023-01-11T21:20:41.4249880Z parm:           NVreg_EnablePCIeGen3:int
2023-01-11T21:20:41.4250291Z parm:           NVreg_EnableMSI:int
2023-01-11T21:20:41.4250691Z parm:           NVreg_TCEBypassMode:int
2023-01-11T21:20:41.4251109Z parm:           NVreg_EnableStreamMemOPs:int
2023-01-11T21:20:41.4251608Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2023-01-11T21:20:41.4252282Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2023-01-11T21:20:41.4252779Z parm:           NVreg_EnableS0ixPowerManagement:int
2023-01-11T21:20:41.4253351Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2023-01-11T21:20:41.4253898Z parm:           NVreg_DynamicPowerManagement:int
2023-01-11T21:20:41.4254476Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2023-01-11T21:20:41.4255002Z parm:           NVreg_EnableGpuFirmware:int
2023-01-11T21:20:41.4255456Z parm:           NVreg_EnableGpuFirmwareLogs:int
2023-01-11T21:20:41.4255955Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2023-01-11T21:20:41.4256448Z parm:           NVreg_EnableUserNUMAManagement:int
2023-01-11T21:20:41.4256901Z parm:           NVreg_MemoryPoolSize:int
2023-01-11T21:20:41.4257340Z parm:           NVreg_KMallocHeapMaxSize:int
2023-01-11T21:20:41.4257786Z parm:           NVreg_VMallocHeapMaxSize:int
2023-01-11T21:20:41.4258226Z parm:           NVreg_IgnoreMMIOCheck:int
2023-01-11T21:20:41.4258651Z parm:           NVreg_NvLinkDisable:int
2023-01-11T21:20:41.4259112Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2023-01-11T21:20:41.4259722Z parm:           NVreg_RegisterPCIDriver:int
2023-01-11T21:20:41.4260188Z parm:           NVreg_EnableDbgBreakpoint:int
2023-01-11T21:20:41.4260612Z parm:           NVreg_RegistryDwords:charp
2023-01-11T21:20:41.4261082Z parm:           NVreg_RegistryDwordsPerDevice:charp
2023-01-11T21:20:41.4261521Z parm:           NVreg_RmMsg:charp
2023-01-11T21:20:41.4261922Z parm:           NVreg_GpuBlacklist:charp
2023-01-11T21:20:41.4262347Z parm:           NVreg_TemporaryFilePath:charp
2023-01-11T21:20:41.4262788Z parm:           NVreg_ExcludedGpus:charp
2023-01-11T21:20:41.4263220Z parm:           NVreg_DmaRemapPeerMmio:int
2023-01-11T21:20:41.4263629Z parm:           rm_firmware_active:charp
2023-01-11T21:20:41.4264003Z + HAS_NVIDIA_DRIVER=0
2023-01-11T21:20:41.4264444Z ++ command -v nvidia-smi
2023-01-11T21:20:41.4264883Z + '[' -x /usr/bin/nvidia-smi ']'
2023-01-11T21:20:41.4265238Z + set +e
2023-01-11T21:20:41.4265798Z ++ nvidia-smi --query-gpu=driver_version --format=csv,noheader --id=0
2023-01-11T21:20:41.9431568Z + INSTALLED_DRIVER_VERSION=515.76
2023-01-11T21:20:41.9431904Z + NVIDIA_SMI_STATUS=0
2023-01-11T21:20:41.9432301Z + '[' 0 -ne 0 ']'
2023-01-11T21:20:41.9432580Z + '[' 515.76 '!=' 515.76 ']'
2023-01-11T21:20:41.9432849Z + HAS_NVIDIA_DRIVER=1
2023-01-11T21:20:41.9433329Z + echo 'NVIDIA driver (515.76) has already been installed. Skipping NVIDIA driver installation'
2023-01-11T21:20:41.9433699Z + set -e
2023-01-11T21:20:41.9433947Z + '[' 1 -eq 0 ']'
2023-01-11T21:20:41.9434303Z NVIDIA driver (515.76) has already been installed. Skipping NVIDIA driver installation
2023-01-11T21:20:41.9434687Z + post_install_nvidia_driver_common
2023-01-11T21:20:41.9437220Z + sudo modprobe nvidia
2023-01-11T21:20:41.9585277Z + echo 'After installing NVIDIA driver'
2023-01-11T21:20:41.9585569Z + lspci
2023-01-11T21:20:41.9585805Z After installing NVIDIA driver
2023-01-11T21:20:41.9786898Z 00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
2023-01-11T21:20:41.9787348Z 00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
2023-01-11T21:20:41.9787766Z 00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
2023-01-11T21:20:41.9788150Z 00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 01)
2023-01-11T21:20:41.9788523Z 00:02.0 VGA compatible controller: Cirrus Logic GD 5446
2023-01-11T21:20:41.9788917Z 00:03.0 Ethernet controller: Amazon.com, Inc. Elastic Network Adapter (ENA)
2023-01-11T21:20:41.9789353Z 00:1d.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2023-01-11T21:20:41.9789768Z 00:1e.0 VGA compatible controller: NVIDIA Corporation GM204GL [Tesla M60] (rev a1)
2023-01-11T21:20:41.9790198Z 00:1f.0 Unassigned class [ff80]: XenSource, Inc. Xen Platform Device (rev 01)
2023-01-11T21:20:41.9790512Z + lsmod
2023-01-11T21:20:41.9808891Z Module                  Size  Used by
2023-01-11T21:20:41.9809373Z nvidia_modeset       1142784  0
2023-01-11T21:20:41.9809642Z nvidia_uvm           1269760  0
2023-01-11T21:20:41.9809882Z veth                   16384  0
2023-01-11T21:20:41.9810189Z nvidia              40808448  27 nvidia_uvm,nvidia_modeset
2023-01-11T21:20:41.9810504Z drm                   425984  1 nvidia
2023-01-11T21:20:41.9810784Z i2c_core               77824  2 nvidia,drm
2023-01-11T21:20:41.9811063Z backlight              16384  1 nvidia_modeset
2023-01-11T21:20:41.9811343Z xt_conntrack           16384  1
2023-01-11T21:20:41.9811615Z ipt_MASQUERADE         16384  1
2023-01-11T21:20:41.9811901Z nf_nat_masquerade_ipv4    16384  1 ipt_MASQUERADE
2023-01-11T21:20:41.9812207Z nf_conntrack_netlink    49152  0
2023-01-11T21:20:41.9812511Z nfnetlink              16384  2 nf_conntrack_netlink
2023-01-11T21:20:41.9812785Z xfrm_user              45056  1
2023-01-11T21:20:41.9813058Z xfrm_algo              16384  1 xfrm_user
2023-01-11T21:20:41.9813329Z xt_addrtype            16384  2
2023-01-11T21:20:41.9813583Z iptable_filter         16384  1
2023-01-11T21:20:41.9813848Z iptable_nat            16384  1
2023-01-11T21:20:41.9814118Z nf_conntrack_ipv4      16384  3
2023-01-11T21:20:41.9814494Z nf_defrag_ipv4         16384  1 nf_conntrack_ipv4
2023-01-11T21:20:41.9814814Z nf_nat_ipv4            16384  1 iptable_nat
2023-01-11T21:20:41.9815140Z nf_nat                 36864  2 nf_nat_masquerade_ipv4,nf_nat_ipv4
2023-01-11T21:20:41.9815616Z nf_conntrack          155648  7 xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_ipv4,nf_nat,ipt_MASQUERADE,nf_nat_ipv4,nf_conntrack_netlink
2023-01-11T21:20:41.9816009Z br_netfilter           24576  0
2023-01-11T21:20:41.9816292Z bridge                172032  1 br_netfilter
2023-01-11T21:20:41.9816570Z stp                    16384  1 bridge
2023-01-11T21:20:41.9816826Z llc                    16384  2 bridge,stp
2023-01-11T21:20:41.9817091Z overlay                86016  0
2023-01-11T21:20:41.9817349Z sunrpc                393216  1
2023-01-11T21:20:41.9817594Z dm_mirror              28672  0
2023-01-11T21:20:41.9817866Z dm_region_hash         20480  1 dm_mirror
2023-01-11T21:20:41.9818195Z dm_log                 20480  2 dm_region_hash,dm_mirror
2023-01-11T21:20:41.9818504Z dm_mod                143360  2 dm_log,dm_mirror
2023-01-11T21:20:41.9818760Z dax                    69632  1 dm_mod
2023-01-11T21:20:41.9819015Z sb_edac                24576  0
2023-01-11T21:20:41.9819274Z crc32_pclmul           16384  0
2023-01-11T21:20:41.9819524Z ghash_clmulni_intel    16384  0
2023-01-11T21:20:41.9819790Z pcbc                   16384  0
2023-01-11T21:20:41.9820048Z aesni_intel           188416  0
2023-01-11T21:20:41.9820305Z aes_x86_64             20480  1 aesni_intel
2023-01-11T21:20:41.9820573Z ata_piix               36864  0
2023-01-11T21:20:41.9820854Z crypto_simd            16384  1 aesni_intel
2023-01-11T21:20:41.9821130Z glue_helper            16384  1 aesni_intel
2023-01-11T21:20:41.9821407Z pcc_cpufreq            16384  0
2023-01-11T21:20:41.9821684Z libata                266240  1 ata_piix
2023-01-11T21:20:41.9822006Z cryptd                 28672  3 crypto_simd,ghash_clmulni_intel,aesni_intel
2023-01-11T21:20:41.9822335Z mousedev               24576  0
2023-01-11T21:20:41.9822604Z scsi_mod              245760  1 libata
2023-01-11T21:20:41.9822852Z psmouse                32768  0
2023-01-11T21:20:41.9823109Z evdev                  20480  3
2023-01-11T21:20:41.9823358Z button                 16384  0
2023-01-11T21:20:41.9823606Z ena                   114688  0
2023-01-11T21:20:41.9823847Z xen_blkfront           49152  2
2023-01-11T21:20:41.9824105Z crc32c_intel           24576  0
2023-01-11T21:20:41.9824364Z autofs4                49152  2
2023-01-11T21:20:41.9824594Z + modinfo nvidia
2023-01-11T21:20:41.9825076Z filename:       /lib/modules/4.14.252-195.483.amzn2.x86_64/kernel/drivers/video/nvidia.ko
2023-01-11T21:20:41.9825430Z firmware:       nvidia/515.76/gsp.bin
2023-01-11T21:20:41.9825753Z alias:          char-major-195-*
2023-01-11T21:20:41.9826104Z version:        515.76
2023-01-11T21:20:41.9826361Z supported:      external
2023-01-11T21:20:41.9826604Z license:        NVIDIA
2023-01-11T21:20:41.9826881Z srcversion:     51FD9DD90150B35351AFFBB
2023-01-11T21:20:41.9827203Z alias:          pci:v000010DEd*sv*sd*bc06sc80i00*
2023-01-11T21:20:41.9827523Z alias:          pci:v000010DEd*sv*sd*bc03sc02i00*
2023-01-11T21:20:41.9827817Z alias:          pci:v000010DEd*sv*sd*bc03sc00i00*
2023-01-11T21:20:41.9828155Z depends:        i2c-core,drm
2023-01-11T21:20:41.9828427Z retpoline:      Y
2023-01-11T21:20:41.9828656Z name:           nvidia
2023-01-11T21:20:41.9829059Z vermagic:       4.14.252-195.483.amzn2.x86_64 SMP mod_unload modversions 
2023-01-11T21:20:41.9829441Z parm:           NvSwitchRegDwords:NvSwitch regkey (charp)
2023-01-11T21:20:41.9829826Z parm:           NvSwitchBlacklist:NvSwitchBlacklist=uuid[,uuid...] (charp)
2023-01-11T21:20:41.9830197Z parm:           NVreg_ResmanDebugLevel:int
2023-01-11T21:20:41.9830500Z parm:           NVreg_RmLogonRC:int
2023-01-11T21:20:41.9830796Z parm:           NVreg_ModifyDeviceFiles:int
2023-01-11T21:20:41.9831107Z parm:           NVreg_DeviceFileUID:int
2023-01-11T21:20:41.9831468Z parm:           NVreg_DeviceFileGID:int
2023-01-11T21:20:41.9831764Z parm:           NVreg_DeviceFileMode:int
2023-01-11T21:20:41.9832130Z parm:           NVreg_InitializeSystemMemoryAllocations:int
2023-01-11T21:20:41.9832507Z parm:           NVreg_UsePageAttributeTable:int
2023-01-11T21:20:41.9832833Z parm:           NVreg_EnablePCIeGen3:int
2023-01-11T21:20:41.9833111Z parm:           NVreg_EnableMSI:int
2023-01-11T21:20:41.9833403Z parm:           NVreg_TCEBypassMode:int
2023-01-11T21:20:41.9833723Z parm:           NVreg_EnableStreamMemOPs:int
2023-01-11T21:20:41.9834068Z parm:           NVreg_RestrictProfilingToAdminUsers:int
2023-01-11T21:20:41.9834463Z parm:           NVreg_PreserveVideoMemoryAllocations:int
2023-01-11T21:20:41.9834842Z parm:           NVreg_EnableS0ixPowerManagement:int
2023-01-11T21:20:41.9835235Z parm:           NVreg_S0ixPowerManagementVideoMemoryThreshold:int
2023-01-11T21:20:41.9835644Z parm:           NVreg_DynamicPowerManagement:int
2023-01-11T21:20:41.9836068Z parm:           NVreg_DynamicPowerManagementVideoMemoryThreshold:int
2023-01-11T21:20:41.9836452Z parm:           NVreg_EnableGpuFirmware:int
2023-01-11T21:20:41.9836806Z parm:           NVreg_EnableGpuFirmwareLogs:int
2023-01-11T21:20:41.9837177Z parm:           NVreg_OpenRmEnableUnsupportedGpus:int
2023-01-11T21:20:41.9837551Z parm:           NVreg_EnableUserNUMAManagement:int
2023-01-11T21:20:41.9837869Z parm:           NVreg_MemoryPoolSize:int
2023-01-11T21:20:41.9838194Z parm:           NVreg_KMallocHeapMaxSize:int
2023-01-11T21:20:41.9838584Z parm:           NVreg_VMallocHeapMaxSize:int
2023-01-11T21:20:41.9839058Z parm:           NVreg_IgnoreMMIOCheck:int
2023-01-11T21:20:41.9839394Z parm:           NVreg_NvLinkDisable:int
2023-01-11T21:20:41.9839838Z parm:           NVreg_EnablePCIERelaxedOrderingMode:int
2023-01-11T21:20:41.9840277Z parm:           NVreg_RegisterPCIDriver:int
2023-01-11T21:20:41.9840633Z parm:           NVreg_EnableDbgBreakpoint:int
2023-01-11T21:20:41.9841069Z parm:           NVreg_RegistryDwords:charp
2023-01-11T21:20:41.9841511Z parm:           NVreg_RegistryDwordsPerDevice:charp
2023-01-11T21:20:41.9841845Z parm:           NVreg_RmMsg:charp
2023-01-11T21:20:41.9842191Z parm:           NVreg_GpuBlacklist:charp
2023-01-11T21:20:41.9842564Z parm:           NVreg_TemporaryFilePath:charp
2023-01-11T21:20:41.9842927Z parm:           NVreg_ExcludedGpus:charp
2023-01-11T21:20:41.9843314Z parm:           NVreg_DmaRemapPeerMmio:int
2023-01-11T21:20:41.9843687Z parm:           rm_firmware_active:charp
2023-01-11T21:20:41.9844009Z + set +e
2023-01-11T21:20:41.9844734Z + nvidia-smi
2023-01-11T21:20:42.0030459Z Wed Jan 11 21:20:41 2023       
2023-01-11T21:20:42.0030989Z +-----------------------------------------------------------------------------+
2023-01-11T21:20:42.0031508Z | NVIDIA-SMI 515.76       Driver Version: 515.76       CUDA Version: 11.7     |
2023-01-11T21:20:42.0032202Z |-------------------------------+----------------------+----------------------+
2023-01-11T21:20:42.0032876Z | GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
2023-01-11T21:20:42.0033462Z | Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
2023-01-11T21:20:42.0033849Z |                               |                      |               MIG M. |
2023-01-11T21:20:42.0034215Z |===============================+======================+======================|
2023-01-11T21:20:42.0095018Z |   0  Tesla M60           Off  | 00000000:00:1D.0 Off |           4294944319 |
2023-01-11T21:20:42.0096117Z | N/A   26C    P0    39W / 150W |      0MiB /  7680MiB |      0%      Default |
2023-01-11T21:20:42.0096842Z |                               |                      |                  N/A |
2023-01-11T21:20:42.0097442Z +-------------------------------+----------------------+----------------------+
2023-01-11T21:20:42.0171948Z |   1  Tesla M60           Off  | 00000000:00:1E.0 Off |          12325814270 |
2023-01-11T21:20:42.0172872Z | N/A   34C    P0    39W / 150W |      0MiB /  7680MiB |     64%      Default |
2023-01-11T21:20:42.0173818Z |                               |                      |                  N/A |
2023-01-11T21:20:42.0174764Z +-------------------------------+----------------------+----------------------+
2023-01-11T21:20:42.0175666Z                                                                                
2023-01-11T21:20:42.0176433Z +-----------------------------------------------------------------------------+
2023-01-11T21:20:42.0176966Z | Processes:                                                                  |
2023-01-11T21:20:42.0177375Z |  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
2023-01-11T21:20:42.0177794Z |        ID   ID                                                   Usage      |
2023-01-11T21:20:42.0178110Z |=============================================================================|
2023-01-11T21:20:42.0184056Z |  No running processes found                                                 |
2023-01-11T21:20:42.0185228Z +-----------------------------------------------------------------------------+
2023-01-11T21:20:42.0741038Z + NVIDIA_SMI_STATUS=0
2023-01-11T21:20:42.0742542Z + '[' 0 -eq 0 ']'
2023-01-11T21:20:42.0743393Z + echo 'INFO: Ignoring allowed status 0'
2023-01-11T21:20:42.0744207Z + set -e
2023-01-11T21:20:42.0744497Z INFO: Ignoring allowed status 0
2023-01-11T21:20:42.0748392Z == Installing nvidia container toolkit for amzn2 ==
2023-01-11T21:20:42.0752405Z + sudo yum install -y yum-utils
2023-01-11T21:20:42.6205655Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2023-01-11T21:20:44.3459332Z Package yum-utils-1.1.31-46.amzn2.0.1.noarch already installed and latest version
2023-01-11T21:20:44.3459909Z Nothing to do
2023-01-11T21:20:44.4308974Z + sudo yum-config-manager --add-repo https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2023-01-11T21:20:45.0193923Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2023-01-11T21:20:45.0519102Z adding repo from: https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo
2023-01-11T21:20:45.0520416Z grabbing file https://nvidia.github.io/nvidia-docker/amzn2/nvidia-docker.repo to /etc/yum.repos.d/nvidia-docker.repo
2023-01-11T21:20:45.0520945Z repo saved to /etc/yum.repos.d/nvidia-docker.repo
2023-01-11T21:20:45.0668889Z + sudo yum install -y nvidia-docker2
2023-01-11T21:20:45.6131284Z Loaded plugins: extras_suggestions, langpacks, priorities, update-motd
2023-01-11T21:20:46.9397327Z Package nvidia-docker2-2.11.0-1.noarch already installed and latest version
2023-01-11T21:20:46.9398052Z Nothing to do
2023-01-11T21:20:47.0176666Z + sudo systemctl restart docker
2023-01-11T21:21:06.7467511Z Command completed after 1 attempt(s).
2023-01-11T21:21:06.7521601Z ##[group]Run python3 -m pip install psutil==5.9.1
2023-01-11T21:21:06.7521997Z python3 -m pip install psutil==5.9.1
2023-01-11T21:21:06.7522455Z python3 -m pip install pynvml==11.4.1
2023-01-11T21:21:06.7522816Z python3 -m tools.stats.monitor > usage_log.txt 2>&1 &
2023-01-11T21:21:06.7523205Z echo "monitor-script-pid=${!}" >> "${GITHUB_OUTPUT}"
2023-01-11T21:21:06.7537059Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:21:06.7537365Z env:
2023-01-11T21:21:06.7537611Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:21:06.7537864Z   GPU_FLAG: --gpus all
2023-01-11T21:21:06.7538119Z ##[endgroup]
2023-01-11T21:21:07.0522847Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T21:21:07.0753350Z Requirement already satisfied: psutil==5.9.1 in /home/ec2-user/.local/lib/python3.7/site-packages (5.9.1)
2023-01-11T21:21:07.6548849Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T21:21:07.6782082Z Requirement already satisfied: pynvml==11.4.1 in /home/ec2-user/.local/lib/python3.7/site-packages (11.4.1)
2023-01-11T21:21:07.9719619Z Prepare all required actions
2023-01-11T21:21:07.9720012Z Getting action download info
2023-01-11T21:21:08.5749059Z Download action repository 'seemethere/download-artifact-s3@v4' (SHA:4a8bfae15cc25cc0785c1603ee87a9da8fd442ea)
2023-01-11T21:21:08.7737061Z Download action repository 'actions/download-artifact@v3' (SHA:9bc31d5ccc31df68ecc42ccf4149144866c47d8a)
2023-01-11T21:21:08.9194123Z ##[group]Run ./.github/actions/download-build-artifacts
2023-01-11T21:21:08.9194434Z with:
2023-01-11T21:21:08.9194699Z   name: linux-bionic-cuda11.7-py3.10-gcc7
2023-01-11T21:21:08.9194981Z env:
2023-01-11T21:21:08.9195218Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:21:08.9195466Z   GPU_FLAG: --gpus all
2023-01-11T21:21:08.9195717Z ##[endgroup]
2023-01-11T21:21:08.9227334Z ##[group]Run seemethere/download-artifact-s3@v4
2023-01-11T21:21:08.9227634Z with:
2023-01-11T21:21:08.9227901Z   name: linux-bionic-cuda11.7-py3.10-gcc7
2023-01-11T21:21:08.9228220Z   s3-bucket: gha-artifacts
2023-01-11T21:21:08.9228545Z   region: us-east-1
2023-01-11T21:21:08.9228768Z env:
2023-01-11T21:21:08.9229008Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:21:08.9229278Z   GPU_FLAG: --gpus all
2023-01-11T21:21:08.9229509Z ##[endgroup]
2023-01-11T21:21:09.4706192Z Found 1 objects with prefix pytorch/pytorch/3896346758/linux-bionic-cuda11.7-py3.10-gcc7/
2023-01-11T21:21:09.4706818Z Starting download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2023-01-11T21:21:15.9186920Z Finished download (1/1): /home/ec2-user/actions-runner/_work/pytorch/pytorch/artifacts.zip
2023-01-11T21:21:15.9187546Z 
2023-01-11T21:21:15.9210080Z ##[warning]The `set-output` command is deprecated and will be disabled soon. Please upgrade to using Environment Files. For more information see: https://github.blog/changelog/2022-10-11-github-actions-deprecating-save-state-and-set-output-commands/
2023-01-11T21:21:15.9220138Z Artifact download has finished successfully
2023-01-11T21:21:15.9490372Z ##[group]Run unzip -o artifacts.zip
2023-01-11T21:21:15.9490718Z unzip -o artifacts.zip
2023-01-11T21:21:15.9504429Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:21:15.9504723Z env:
2023-01-11T21:21:15.9504979Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:21:15.9505263Z   GPU_FLAG: --gpus all
2023-01-11T21:21:15.9505505Z ##[endgroup]
2023-01-11T21:21:15.9548964Z Archive:  artifacts.zip
2023-01-11T21:21:15.9550989Z    creating: dist/
2023-01-11T21:21:18.0408622Z   inflating: dist/torch-2.0.0a0+git8419ddd-cp310-cp310-linux_x86_64.whl  
2023-01-11T21:21:18.0409053Z    creating: build/custom_test_artifacts/
2023-01-11T21:21:18.0409481Z    creating: build/custom_test_artifacts/custom-op-build/
2023-01-11T21:21:18.0409941Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/
2023-01-11T21:21:18.0417030Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeOutput.log  
2023-01-11T21:21:18.0417587Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/
2023-01-11T21:21:18.0418416Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2023-01-11T21:21:18.0418964Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/
2023-01-11T21:21:18.0419526Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2023-01-11T21:21:18.0421815Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2023-01-11T21:21:18.0423169Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2023-01-11T21:21:18.0423726Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/
2023-01-11T21:21:18.0424297Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2023-01-11T21:21:18.0426614Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-01-11T21:21:18.0427828Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2023-01-11T21:21:18.0429533Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2023-01-11T21:21:18.0430172Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2023-01-11T21:21:18.0431946Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2023-01-11T21:21:18.0432841Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2023-01-11T21:21:18.0433418Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2023-01-11T21:21:18.0433998Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2023-01-11T21:21:18.0488688Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2023-01-11T21:21:18.0489428Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2023-01-11T21:21:18.0490148Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2023-01-11T21:21:18.0490903Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2023-01-11T21:21:18.0491643Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2023-01-11T21:21:18.0492350Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2023-01-11T21:21:18.0493058Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2023-01-11T21:21:18.0493749Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2023-01-11T21:21:18.0494669Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2023-01-11T21:21:18.0536980Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2023-01-11T21:21:18.0578492Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2023-01-11T21:21:18.0579588Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2023-01-11T21:21:18.0580563Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2023-01-11T21:21:18.0581234Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2023-01-11T21:21:18.0582078Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2023-01-11T21:21:18.0582914Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2023-01-11T21:21:18.0583879Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2023-01-11T21:21:18.0585915Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2023-01-11T21:21:18.0659574Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2023-01-11T21:21:18.0732755Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2023-01-11T21:21:18.0733631Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2023-01-11T21:21:18.0734216Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeTmp/
2023-01-11T21:21:18.0735072Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeError.log  
2023-01-11T21:21:18.0735662Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/cmake.check_cache  
2023-01-11T21:21:18.0736209Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/
2023-01-11T21:21:18.0736797Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.ts  
2023-01-11T21:21:18.0737405Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/compiler_depend.make  
2023-01-11T21:21:18.0738016Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/depend.make  
2023-01-11T21:21:18.0738605Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/link.txt  
2023-01-11T21:21:18.0739201Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/cmake_clean.cmake  
2023-01-11T21:21:18.0740009Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/build.make  
2023-01-11T21:21:18.0740884Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/DependInfo.cmake  
2023-01-11T21:21:18.0741511Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/flags.make  
2023-01-11T21:21:18.0742120Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/progress.make  
2023-01-11T21:21:18.0763308Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o.d  
2023-01-11T21:21:18.0880825Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/custom_ops.dir/op.cpp.o  
2023-01-11T21:21:18.0881396Z    creating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/
2023-01-11T21:21:18.0882004Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.ts  
2023-01-11T21:21:18.0882669Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/compiler_depend.make  
2023-01-11T21:21:18.0883300Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/depend.make  
2023-01-11T21:21:18.0883893Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/link.txt  
2023-01-11T21:21:18.0884750Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/cmake_clean.cmake  
2023-01-11T21:21:18.0885602Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/build.make  
2023-01-11T21:21:18.0886459Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/DependInfo.cmake  
2023-01-11T21:21:18.0887099Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/flags.make  
2023-01-11T21:21:18.0887724Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/progress.make  
2023-01-11T21:21:18.0908986Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o.d  
2023-01-11T21:21:18.0996365Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/test_custom_ops.dir/test_custom_ops.cpp.o  
2023-01-11T21:21:18.0997022Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-01-11T21:21:18.0997644Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/TargetDirectories.txt  
2023-01-11T21:21:18.0998217Z  extracting: build/custom_test_artifacts/custom-op-build/CMakeFiles/progress.marks  
2023-01-11T21:21:18.0998979Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile2  
2023-01-11T21:21:18.1000235Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeFiles/Makefile.cmake  
2023-01-11T21:21:18.1000777Z   inflating: build/custom_test_artifacts/custom-op-build/detect_cuda_version.cc  
2023-01-11T21:21:18.1003873Z   inflating: build/custom_test_artifacts/custom-op-build/CMakeCache.txt  
2023-01-11T21:21:18.1004745Z   inflating: build/custom_test_artifacts/custom-op-build/Makefile  
2023-01-11T21:21:18.1005433Z   inflating: build/custom_test_artifacts/custom-op-build/cmake_install.cmake  
2023-01-11T21:21:18.1100546Z   inflating: build/custom_test_artifacts/custom-op-build/libcustom_ops.so  
2023-01-11T21:21:18.1165725Z   inflating: build/custom_test_artifacts/custom-op-build/test_custom_ops  
2023-01-11T21:21:18.1166208Z    creating: build/custom_test_artifacts/jit-hook-build/
2023-01-11T21:21:18.1166653Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/
2023-01-11T21:21:18.1173380Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeOutput.log  
2023-01-11T21:21:18.1173923Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/
2023-01-11T21:21:18.1174473Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2023-01-11T21:21:18.1175035Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/
2023-01-11T21:21:18.1175594Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2023-01-11T21:21:18.1177655Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2023-01-11T21:21:18.1178742Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2023-01-11T21:21:18.1179288Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/
2023-01-11T21:21:18.1179862Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2023-01-11T21:21:18.1182676Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-01-11T21:21:18.1183750Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2023-01-11T21:21:18.1185709Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2023-01-11T21:21:18.1186344Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2023-01-11T21:21:18.1187796Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2023-01-11T21:21:18.1188937Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2023-01-11T21:21:18.1189504Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2023-01-11T21:21:18.1190079Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2023-01-11T21:21:18.1244508Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2023-01-11T21:21:18.1245508Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2023-01-11T21:21:18.1246447Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2023-01-11T21:21:18.1247175Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2023-01-11T21:21:18.1247952Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2023-01-11T21:21:18.1248654Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2023-01-11T21:21:18.1249357Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2023-01-11T21:21:18.1250032Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2023-01-11T21:21:18.1250837Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2023-01-11T21:21:18.1292748Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2023-01-11T21:21:18.1333934Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2023-01-11T21:21:18.1334929Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2023-01-11T21:21:18.1335761Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2023-01-11T21:21:18.1336601Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2023-01-11T21:21:18.1337478Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2023-01-11T21:21:18.1338284Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2023-01-11T21:21:18.1339305Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2023-01-11T21:21:18.1341356Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2023-01-11T21:21:18.1414857Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2023-01-11T21:21:18.1489952Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2023-01-11T21:21:18.1490602Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2023-01-11T21:21:18.1491143Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeTmp/
2023-01-11T21:21:18.1491936Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeError.log  
2023-01-11T21:21:18.1492637Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/cmake.check_cache  
2023-01-11T21:21:18.1493216Z    creating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/
2023-01-11T21:21:18.1493812Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.ts  
2023-01-11T21:21:18.1494430Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/compiler_depend.make  
2023-01-11T21:21:18.1495042Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/depend.make  
2023-01-11T21:21:18.1495778Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/link.txt  
2023-01-11T21:21:18.1496384Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/cmake_clean.cmake  
2023-01-11T21:21:18.1497772Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/build.make  
2023-01-11T21:21:18.1498576Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/DependInfo.cmake  
2023-01-11T21:21:18.1499346Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/flags.make  
2023-01-11T21:21:18.1499937Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/progress.make  
2023-01-11T21:21:18.1521014Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o.d  
2023-01-11T21:21:18.1588271Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/test_jit_hooks.dir/test_jit_hooks.cpp.o  
2023-01-11T21:21:18.1588911Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-01-11T21:21:18.1589505Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/TargetDirectories.txt  
2023-01-11T21:21:18.1590073Z  extracting: build/custom_test_artifacts/jit-hook-build/CMakeFiles/progress.marks  
2023-01-11T21:21:18.1590907Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile2  
2023-01-11T21:21:18.1591977Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeFiles/Makefile.cmake  
2023-01-11T21:21:18.1592500Z   inflating: build/custom_test_artifacts/jit-hook-build/detect_cuda_version.cc  
2023-01-11T21:21:18.1595495Z   inflating: build/custom_test_artifacts/jit-hook-build/CMakeCache.txt  
2023-01-11T21:21:18.1596039Z   inflating: build/custom_test_artifacts/jit-hook-build/Makefile  
2023-01-11T21:21:18.1596991Z   inflating: build/custom_test_artifacts/jit-hook-build/cmake_install.cmake  
2023-01-11T21:21:18.1648387Z   inflating: build/custom_test_artifacts/jit-hook-build/test_jit_hooks  
2023-01-11T21:21:18.1648885Z    creating: build/custom_test_artifacts/custom-backend-build/
2023-01-11T21:21:18.1649383Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/
2023-01-11T21:21:18.1656836Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeOutput.log  
2023-01-11T21:21:18.1657407Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/
2023-01-11T21:21:18.1657991Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeSystem.cmake  
2023-01-11T21:21:18.1658585Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/
2023-01-11T21:21:18.1659168Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/tmp/
2023-01-11T21:21:18.1661223Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/CMakeCCompilerId.c  
2023-01-11T21:21:18.1662364Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdC/a.out  
2023-01-11T21:21:18.1662965Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/
2023-01-11T21:21:18.1663565Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/tmp/
2023-01-11T21:21:18.1666262Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/CMakeCXXCompilerId.cpp  
2023-01-11T21:21:18.1667364Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCXX/a.out  
2023-01-11T21:21:18.1669523Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_C.bin  
2023-01-11T21:21:18.1670178Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCCompiler.cmake  
2023-01-11T21:21:18.1671653Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CXX.bin  
2023-01-11T21:21:18.1672793Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCXXCompiler.cmake  
2023-01-11T21:21:18.1673417Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/
2023-01-11T21:21:18.1674014Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/
2023-01-11T21:21:18.1728705Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp1.ii  
2023-01-11T21:21:18.1729450Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.c  
2023-01-11T21:21:18.1730215Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.gpu  
2023-01-11T21:21:18.1730990Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.stub.c  
2023-01-11T21:21:18.1731745Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.module_id  
2023-01-11T21:21:18.1732479Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.ptx  
2023-01-11T21:21:18.1733307Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.sm_52.cubin  
2023-01-11T21:21:18.1734071Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin  
2023-01-11T21:21:18.1734809Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.fatbin.c  
2023-01-11T21:21:18.1777525Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cpp4.ii  
2023-01-11T21:21:18.1818696Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.cudafe1.cpp  
2023-01-11T21:21:18.1819864Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/CMakeCUDACompilerId.o  
2023-01-11T21:21:18.1820721Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.sm_52.cubin  
2023-01-11T21:21:18.1821390Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.reg.c  
2023-01-11T21:21:18.1822214Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin  
2023-01-11T21:21:18.1823021Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.fatbin.c  
2023-01-11T21:21:18.1824117Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/tmp/a_dlink.o  
2023-01-11T21:21:18.1826166Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/CMakeCUDACompilerId.cu  
2023-01-11T21:21:18.1899740Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CompilerIdCUDA/a.out  
2023-01-11T21:21:18.1972984Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeDetermineCompilerABI_CUDA.bin  
2023-01-11T21:21:18.1973674Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/3.22.1/CMakeCUDACompiler.cmake  
2023-01-11T21:21:18.1974264Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeTmp/
2023-01-11T21:21:18.1975012Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeError.log  
2023-01-11T21:21:18.1975576Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/cmake.check_cache  
2023-01-11T21:21:18.1976162Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/
2023-01-11T21:21:18.1976793Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.ts  
2023-01-11T21:21:18.1977463Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/compiler_depend.make  
2023-01-11T21:21:18.1978108Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/depend.make  
2023-01-11T21:21:18.1978851Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/link.txt  
2023-01-11T21:21:18.1979485Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/cmake_clean.cmake  
2023-01-11T21:21:18.1980311Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/build.make  
2023-01-11T21:21:18.1980966Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/DependInfo.cmake  
2023-01-11T21:21:18.1981786Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/flags.make  
2023-01-11T21:21:18.1982453Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/progress.make  
2023-01-11T21:21:18.1987146Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o.d  
2023-01-11T21:21:18.2140670Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/custom_backend.dir/custom_backend.cpp.o  
2023-01-11T21:21:18.2141336Z    creating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/
2023-01-11T21:21:18.2141987Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.ts  
2023-01-11T21:21:18.2142674Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/compiler_depend.make  
2023-01-11T21:21:18.2143331Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/depend.make  
2023-01-11T21:21:18.2143962Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/link.txt  
2023-01-11T21:21:18.2144613Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/cmake_clean.cmake  
2023-01-11T21:21:18.2145273Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/build.make  
2023-01-11T21:21:18.2145937Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/DependInfo.cmake  
2023-01-11T21:21:18.2146567Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/flags.make  
2023-01-11T21:21:18.2147538Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/progress.make  
2023-01-11T21:21:18.2169209Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o.d  
2023-01-11T21:21:18.2230557Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/test_custom_backend.dir/test_custom_backend.cpp.o  
2023-01-11T21:21:18.2231228Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/CMakeDirectoryInformation.cmake  
2023-01-11T21:21:18.2231888Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/TargetDirectories.txt  
2023-01-11T21:21:18.2232493Z  extracting: build/custom_test_artifacts/custom-backend-build/CMakeFiles/progress.marks  
2023-01-11T21:21:18.2233077Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile2  
2023-01-11T21:21:18.2234229Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeFiles/Makefile.cmake  
2023-01-11T21:21:18.2234800Z   inflating: build/custom_test_artifacts/custom-backend-build/detect_cuda_version.cc  
2023-01-11T21:21:18.2237738Z   inflating: build/custom_test_artifacts/custom-backend-build/CMakeCache.txt  
2023-01-11T21:21:18.2238464Z   inflating: build/custom_test_artifacts/custom-backend-build/Makefile  
2023-01-11T21:21:18.2239288Z   inflating: build/custom_test_artifacts/custom-backend-build/cmake_install.cmake  
2023-01-11T21:21:18.2363042Z   inflating: build/custom_test_artifacts/custom-backend-build/libcustom_backend.so  
2023-01-11T21:21:18.2411172Z   inflating: build/custom_test_artifacts/custom-backend-build/test_custom_backend  
2023-01-11T21:21:18.2411679Z    creating: build/lib/
2023-01-11T21:21:18.2412135Z   inflating: build/lib/libclog.a     
2023-01-11T21:21:18.2481969Z   inflating: build/lib/libgtest.a    
2023-01-11T21:21:18.2492323Z   inflating: build/lib/libpthreadpool.a  
2023-01-11T21:21:18.2588841Z   inflating: build/lib/libbenchmark.a  
2023-01-11T21:21:18.2598063Z   inflating: build/lib/libittnotify.a  
2023-01-11T21:21:18.2704744Z   inflating: build/lib/libprotobuf-lite.a  
2023-01-11T21:21:18.2736769Z   inflating: build/lib/libtensorpipe_uv.a  
2023-01-11T21:21:18.2813704Z   inflating: build/lib/libasmjit.a   
2023-01-11T21:21:18.3348108Z   inflating: build/lib/libprotobuf.a  
2023-01-11T21:21:18.3489558Z   inflating: build/lib/libgloo.a     
2023-01-11T21:21:18.3521917Z   inflating: build/lib/libfmt.a      
2023-01-11T21:21:18.3522572Z   inflating: build/lib/libfoxi_loader.a  
2023-01-11T21:21:18.3524825Z   inflating: build/lib/libcaffe2_nvrtc.so  
2023-01-11T21:21:18.3607646Z   inflating: build/lib/libc10.so     
2023-01-11T21:21:18.3608898Z   inflating: build/lib/libtorch_global_deps.so  
2023-01-11T21:21:18.4179820Z   inflating: build/lib/libprotoc.a   
2023-01-11T21:21:18.4189349Z   inflating: build/lib/libcpuinfo.a  
2023-01-11T21:21:18.4191980Z   inflating: build/lib/libnnpack_reference_layers.a  
2023-01-11T21:21:18.4200819Z   inflating: build/lib/libcpuinfo_internals.a  
2023-01-11T21:21:18.4218746Z   inflating: build/lib/libgmock.a    
2023-01-11T21:21:18.4219251Z   inflating: build/lib/libgtest_main.a  
2023-01-11T21:21:18.4220323Z   inflating: build/lib/libbenchmark_main.a  
2023-01-11T21:21:18.4877300Z   inflating: build/lib/libtensorpipe.a  
2023-01-11T21:21:19.4650559Z   inflating: build/lib/libdnnl.a     
2023-01-11T21:21:19.4791989Z   inflating: build/lib/libXNNPACK.a  
2023-01-11T21:21:19.4846762Z   inflating: build/lib/libc10_cuda.so  
2023-01-11T21:21:19.4862722Z   inflating: build/lib/libqnnpack.a  
2023-01-11T21:21:19.4863248Z   inflating: build/lib/libgmock_main.a  
2023-01-11T21:21:19.6406073Z   inflating: build/lib/libfbgemm.a   
2023-01-11T21:21:19.6428914Z   inflating: build/lib/libpytorch_qnnpack.a  
2023-01-11T21:21:19.7585572Z   inflating: build/lib/libdnnl_graph.a  
2023-01-11T21:21:19.8102060Z   inflating: build/lib/libkineto.a   
2023-01-11T21:21:19.8392370Z   inflating: build/lib/libtensorpipe_cuda.a  
2023-01-11T21:21:19.8437524Z   inflating: build/lib/libcaffe2_protos.a  
2023-01-11T21:21:19.8459887Z   inflating: build/lib/libnnpack.a   
2023-01-11T21:21:19.8508370Z   inflating: build/lib/libonnx_proto.a  
2023-01-11T21:21:19.9187059Z   inflating: build/lib/libonnx.a     
2023-01-11T21:21:19.9619323Z   inflating: build/lib/libgloo_cuda.a  
2023-01-11T21:21:22.3450892Z   inflating: build/lib/libtorch_cpu.so  
2023-01-11T21:21:22.3461448Z   inflating: build/lib/libunbox_lib.a  
2023-01-11T21:21:24.4728586Z   inflating: build/lib/libtorch_cuda.so  
2023-01-11T21:21:24.4729426Z   inflating: build/lib/libtorch.so   
2023-01-11T21:21:24.4732642Z   inflating: build/lib/libc10d_cuda_test.so  
2023-01-11T21:21:25.4628770Z   inflating: build/lib/libtorch_cuda_linalg.so  
2023-01-11T21:21:25.4690108Z   inflating: build/lib/libtorchbind_test.so  
2023-01-11T21:21:25.4715082Z   inflating: build/lib/libjitbackend_test.so  
2023-01-11T21:21:25.4746165Z   inflating: build/lib/libbackend_with_compiler.so  
2023-01-11T21:21:25.4751144Z   inflating: build/lib/libshm.so     
2023-01-11T21:21:25.6605464Z   inflating: build/lib/libtorch_python.so  
2023-01-11T21:21:25.6645473Z   inflating: build/lib/libnnapi_backend.so  
2023-01-11T21:21:25.6645793Z    creating: build/bin/
2023-01-11T21:21:25.6700354Z   inflating: build/bin/c10_CompileTimeFunctionPointer_test  
2023-01-11T21:21:25.6757786Z   inflating: build/bin/c10_DeviceGuard_test  
2023-01-11T21:21:25.6814150Z   inflating: build/bin/c10_Device_test  
2023-01-11T21:21:25.6878755Z   inflating: build/bin/c10_DispatchKeySet_test  
2023-01-11T21:21:25.6932196Z   inflating: build/bin/c10_StreamGuard_test  
2023-01-11T21:21:25.6986681Z   inflating: build/bin/c10_SymInt_test  
2023-01-11T21:21:25.7048506Z   inflating: build/bin/c10_InlineDeviceGuard_test  
2023-01-11T21:21:25.7110213Z   inflating: build/bin/c10_InlineStreamGuard_test  
2023-01-11T21:21:25.7173305Z   inflating: build/bin/c10_SizesAndStrides_test  
2023-01-11T21:21:25.7226486Z   inflating: build/bin/c10_Array_test  
2023-01-11T21:21:25.7285559Z   inflating: build/bin/c10_Bitset_test  
2023-01-11T21:21:25.7342326Z   inflating: build/bin/c10_C++17_test  
2023-01-11T21:21:25.7395871Z   inflating: build/bin/c10_ConstexprCrc_test  
2023-01-11T21:21:25.7450380Z   inflating: build/bin/c10_DeadlockDetection_test  
2023-01-11T21:21:25.7505394Z   inflating: build/bin/c10_Half_test  
2023-01-11T21:21:25.7568285Z   inflating: build/bin/c10_LeftRight_test  
2023-01-11T21:21:25.7637410Z   inflating: build/bin/c10_Metaprogramming_test  
2023-01-11T21:21:25.7693149Z   inflating: build/bin/c10_Synchronized_test  
2023-01-11T21:21:25.7853960Z   inflating: build/bin/c10_SmallVectorTest  
2023-01-11T21:21:25.7916950Z   inflating: build/bin/c10_ThreadLocal_test  
2023-01-11T21:21:25.7975726Z   inflating: build/bin/c10_TypeIndex_test  
2023-01-11T21:21:25.8031835Z   inflating: build/bin/c10_TypeList_test  
2023-01-11T21:21:25.8085031Z   inflating: build/bin/c10_TypeTraits_test  
2023-01-11T21:21:25.8142960Z   inflating: build/bin/c10_accumulate_test  
2023-01-11T21:21:25.8205582Z   inflating: build/bin/c10_bfloat16_test  
2023-01-11T21:21:25.8266836Z   inflating: build/bin/c10_complex_math_test  
2023-01-11T21:21:25.8327969Z   inflating: build/bin/c10_complex_test  
2023-01-11T21:21:25.8447853Z   inflating: build/bin/c10_either_test  
2023-01-11T21:21:25.8506703Z   inflating: build/bin/c10_exception_test  
2023-01-11T21:21:25.8562124Z   inflating: build/bin/c10_flags_test  
2023-01-11T21:21:25.8746748Z   inflating: build/bin/c10_intrusive_ptr_test  
2023-01-11T21:21:25.8802685Z   inflating: build/bin/c10_irange_test  
2023-01-11T21:21:25.8866018Z   inflating: build/bin/c10_logging_test  
2023-01-11T21:21:25.8947614Z   inflating: build/bin/c10_optional_test  
2023-01-11T21:21:25.9016974Z   inflating: build/bin/c10_ordered_preserving_dict_test  
2023-01-11T21:21:25.9077998Z   inflating: build/bin/c10_registry_test  
2023-01-11T21:21:25.9142616Z   inflating: build/bin/c10_string_view_test  
2023-01-11T21:21:25.9200058Z   inflating: build/bin/c10_tempfile_test  
2023-01-11T21:21:25.9262017Z   inflating: build/bin/c10_typeid_test  
2023-01-11T21:21:25.9323099Z   inflating: build/bin/c10_intrusive_ptr_benchmark  
2023-01-11T21:21:25.9845275Z   inflating: build/bin/protoc-3.13.0.0  
2023-01-11T21:21:26.0367305Z   inflating: build/bin/protoc        
2023-01-11T21:21:26.0426677Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_stream  
2023-01-11T21:21:26.0485773Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_1_var_test  
2023-01-11T21:21:26.0544460Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_catches_thread_and_block_and_device  
2023-01-11T21:21:26.0602406Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_from_2_processes  
2023-01-11T21:21:26.0661638Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_blocks_and_threads  
2023-01-11T21:21:26.0720820Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_multiple_blocks  
2023-01-11T21:21:26.0774676Z   inflating: build/bin/c10_cuda_CUDATest  
2023-01-11T21:21:26.0833547Z   inflating: build/bin/c10_cuda_CUDAAssertionsTest_multiple_writes_from_same_block  
2023-01-11T21:21:26.1157414Z   inflating: build/bin/vec_test_all_types_DEFAULT  
2023-01-11T21:21:26.1517598Z   inflating: build/bin/vec_test_all_types_AVX2  
2023-01-11T21:21:26.1577600Z   inflating: build/bin/HashStoreTest  
2023-01-11T21:21:26.1643545Z   inflating: build/bin/TCPStoreTest  
2023-01-11T21:21:26.1703473Z   inflating: build/bin/FileStoreTest  
2023-01-11T21:21:26.1719929Z   inflating: build/bin/ProcessGroupMPITest  
2023-01-11T21:21:26.1783050Z   inflating: build/bin/test_edge_op_registration  
2023-01-11T21:21:26.1786401Z   inflating: build/bin/example_allreduce  
2023-01-11T21:21:26.1845218Z   inflating: build/bin/Dimname_test  
2023-01-11T21:21:26.1925938Z   inflating: build/bin/Dict_test     
2023-01-11T21:21:26.1996839Z   inflating: build/bin/MaybeOwned_test  
2023-01-11T21:21:26.2060044Z   inflating: build/bin/NamedTensor_test  
2023-01-11T21:21:26.2125848Z   inflating: build/bin/apply_utils_test  
2023-01-11T21:21:26.2193061Z   inflating: build/bin/basic         
2023-01-11T21:21:26.2258233Z   inflating: build/bin/atest         
2023-01-11T21:21:26.2318232Z   inflating: build/bin/broadcast_test  
2023-01-11T21:21:26.2382536Z   inflating: build/bin/cpu_generator_test  
2023-01-11T21:21:26.2440749Z   inflating: build/bin/cpu_profiling_allocator_test  
2023-01-11T21:21:26.2537914Z   inflating: build/bin/cpu_rng_test  
2023-01-11T21:21:26.2593137Z   inflating: build/bin/dispatch_key_set_test  
2023-01-11T21:21:26.2648080Z   inflating: build/bin/dlconvertor_test  
2023-01-11T21:21:26.2712511Z   inflating: build/bin/extension_backend_test  
2023-01-11T21:21:26.2774297Z   inflating: build/bin/half_test     
2023-01-11T21:21:26.2828680Z   inflating: build/bin/lazy_tensor_test  
2023-01-11T21:21:26.2889021Z   inflating: build/bin/math_kernel_test  
2023-01-11T21:21:26.2992704Z   inflating: build/bin/ivalue_test   
2023-01-11T21:21:26.3052595Z   inflating: build/bin/memory_format_test  
2023-01-11T21:21:26.3112216Z   inflating: build/bin/memory_overlapping_test  
2023-01-11T21:21:26.3168632Z   inflating: build/bin/operator_name_test  
2023-01-11T21:21:26.3226855Z   inflating: build/bin/mobile_memory_cleanup  
2023-01-11T21:21:26.3288612Z   inflating: build/bin/native_test   
2023-01-11T21:21:26.3344068Z   inflating: build/bin/operators_test  
2023-01-11T21:21:26.3402875Z   inflating: build/bin/packedtensoraccessor_test  
2023-01-11T21:21:26.3466800Z   inflating: build/bin/quantized_test  
2023-01-11T21:21:26.3539836Z   inflating: build/bin/pow_test      
2023-01-11T21:21:26.3594276Z   inflating: build/bin/reduce_ops_test  
2023-01-11T21:21:26.3650898Z   inflating: build/bin/reportMemoryUsage_test  
2023-01-11T21:21:26.3713478Z   inflating: build/bin/scalar_tensor_test  
2023-01-11T21:21:26.3776934Z   inflating: build/bin/scalar_test   
2023-01-11T21:21:26.3835490Z   inflating: build/bin/stride_properties_test  
2023-01-11T21:21:26.3922025Z   inflating: build/bin/tensor_iterator_test  
2023-01-11T21:21:26.3983116Z   inflating: build/bin/type_ptr_test  
2023-01-11T21:21:26.3986443Z   inflating: build/bin/thread_init_test  
2023-01-11T21:21:26.4048639Z   inflating: build/bin/test_parallel  
2023-01-11T21:21:26.4103276Z   inflating: build/bin/variant_test  
2023-01-11T21:21:26.4161744Z   inflating: build/bin/undefined_tensor_test  
2023-01-11T21:21:26.4229090Z   inflating: build/bin/type_test     
2023-01-11T21:21:26.4230571Z   inflating: build/bin/verify_api_visibility  
2023-01-11T21:21:26.4308124Z   inflating: build/bin/legacy_vmap_test  
2023-01-11T21:21:26.4364788Z   inflating: build/bin/weakref_test  
2023-01-11T21:21:26.4421391Z   inflating: build/bin/wrapdim_test  
2023-01-11T21:21:26.4541414Z   inflating: build/bin/List_test     
2023-01-11T21:21:26.4607903Z   inflating: build/bin/IListRef_test  
2023-01-11T21:21:26.4661962Z   inflating: build/bin/xla_tensor_test  
2023-01-11T21:21:26.4795508Z   inflating: build/bin/kernel_function_legacy_test  
2023-01-11T21:21:26.4867322Z   inflating: build/bin/KernelFunction_test  
2023-01-11T21:21:26.4973165Z   inflating: build/bin/kernel_function_test  
2023-01-11T21:21:26.5114234Z   inflating: build/bin/kernel_lambda_legacy_test  
2023-01-11T21:21:26.5181213Z   inflating: build/bin/kernel_stackbased_test  
2023-01-11T21:21:26.5294823Z   inflating: build/bin/kernel_lambda_test  
2023-01-11T21:21:26.5351634Z   inflating: build/bin/CppSignature_test  
2023-01-11T21:21:26.5456967Z   inflating: build/bin/make_boxed_from_unboxed_functor_test  
2023-01-11T21:21:26.5510028Z   inflating: build/bin/op_allowlist_test  
2023-01-11T21:21:26.5570385Z   inflating: build/bin/inline_container_test  
2023-01-11T21:21:26.5632466Z   inflating: build/bin/backend_fallback_test  
2023-01-11T21:21:26.5947758Z   inflating: build/bin/op_registration_test  
2023-01-11T21:21:26.6006280Z   inflating: build/bin/cuda_apply_test  
2023-01-11T21:21:26.6084170Z   inflating: build/bin/cuda_complex_math_test  
2023-01-11T21:21:26.6138719Z   inflating: build/bin/cuda_device_test  
2023-01-11T21:21:26.6198569Z   inflating: build/bin/cuda_caching_host_allocator_test  
2023-01-11T21:21:26.6264676Z   inflating: build/bin/cuda_atomic_ops_test  
2023-01-11T21:21:26.6319999Z   inflating: build/bin/cuda_dlconvertor_test  
2023-01-11T21:21:26.6384653Z   inflating: build/bin/cuda_complex_test  
2023-01-11T21:21:26.6450553Z   inflating: build/bin/cuda_cub_test  
2023-01-11T21:21:26.6506562Z   inflating: build/bin/cuda_integer_divider_test  
2023-01-11T21:21:26.6580836Z   inflating: build/bin/cuda_distributions_test  
2023-01-11T21:21:26.6639466Z   inflating: build/bin/cuda_reportMemoryUsage_test  
2023-01-11T21:21:26.6706747Z   inflating: build/bin/cuda_stream_test  
2023-01-11T21:21:26.6772096Z   inflating: build/bin/cuda_generator_test  
2023-01-11T21:21:26.6826024Z   inflating: build/bin/cuda_optional_test  
2023-01-11T21:21:26.6880446Z   inflating: build/bin/cuda_half_test  
2023-01-11T21:21:26.6937505Z   inflating: build/bin/cuda_packedtensoraccessor_test  
2023-01-11T21:21:26.6955437Z   inflating: build/bin/tutorial_tensorexpr  
2023-01-11T21:21:26.7027730Z   inflating: build/bin/ProcessGroupGlooTest  
2023-01-11T21:21:26.7081434Z   inflating: build/bin/cuda_cudnn_test  
2023-01-11T21:21:26.7140495Z   inflating: build/bin/ProcessGroupUCCTest  
2023-01-11T21:21:26.7200292Z   inflating: build/bin/test_dist_autograd  
2023-01-11T21:21:26.7264899Z   inflating: build/bin/ProcessGroupGlooAsyncTest  
2023-01-11T21:21:26.7329361Z   inflating: build/bin/ProcessGroupNCCLErrorsTest  
2023-01-11T21:21:26.7397228Z   inflating: build/bin/ProcessGroupNCCLTest  
2023-01-11T21:21:26.7474359Z   inflating: build/bin/test_cpp_rpc  
2023-01-11T21:21:26.7477017Z   inflating: build/bin/parallel_benchmark  
2023-01-11T21:21:26.7552747Z   inflating: build/bin/test_mobile_nnc  
2023-01-11T21:21:26.7564166Z   inflating: build/bin/aot_model_compiler_test  
2023-01-11T21:21:26.7622569Z   inflating: build/bin/cuda_vectorized_test  
2023-01-11T21:21:26.7628412Z   inflating: build/bin/torch_shm_manager  
2023-01-11T21:21:26.8017767Z   inflating: build/bin/test_lazy     
2023-01-11T21:21:26.8928165Z   inflating: build/bin/test_tensorexpr  
2023-01-11T21:21:27.0249653Z   inflating: build/bin/test_api      
2023-01-11T21:21:27.1454420Z   inflating: build/bin/test_jit      
2023-01-11T21:21:27.1457283Z   inflating: .pytorch-test-times.json  
2023-01-11T21:21:27.1487168Z ##[group]Run df -H
2023-01-11T21:21:27.1487411Z df -H
2023-01-11T21:21:27.1500817Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T21:21:27.1501116Z env:
2023-01-11T21:21:27.1501361Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:21:27.1501615Z   GPU_FLAG: --gpus all
2023-01-11T21:21:27.1501874Z ##[endgroup]
2023-01-11T21:21:27.1540836Z Filesystem      Size  Used Avail Use% Mounted on
2023-01-11T21:21:27.1541169Z devtmpfs        129G     0  129G   0% /dev
2023-01-11T21:21:27.1541474Z tmpfs           129G   13M  129G   1% /dev/shm
2023-01-11T21:21:27.1541740Z tmpfs           129G  553k  129G   1% /run
2023-01-11T21:21:27.1542027Z tmpfs           129G     0  129G   0% /sys/fs/cgroup
2023-01-11T21:21:27.1542318Z /dev/xvda1      162G   27G  135G  17% /
2023-01-11T21:21:27.1566878Z ##[group]Run .github/scripts/parse_ref.py
2023-01-11T21:21:27.1567253Z .github/scripts/parse_ref.py
2023-01-11T21:21:27.1579289Z shell: /usr/bin/bash -e {0}
2023-01-11T21:21:27.1579543Z env:
2023-01-11T21:21:27.1579766Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:21:27.1580038Z   GPU_FLAG: --gpus all
2023-01-11T21:21:27.1580290Z ##[endgroup]
2023-01-11T21:21:27.1872150Z ##[group]Run set -x
2023-01-11T21:21:27.1872530Z set -x
2023-01-11T21:21:27.1872764Z 
2023-01-11T21:21:27.1873019Z if [[ $TEST_CONFIG == 'multigpu' ]]; then
2023-01-11T21:21:27.1873501Z   TEST_COMMAND=.jenkins/pytorch/multigpu-test.sh
2023-01-11T21:21:27.1873852Z elif [[ $BUILD_ENVIRONMENT == *onnx* ]]; then
2023-01-11T21:21:27.1874179Z   TEST_COMMAND=.jenkins/onnx/test.sh
2023-01-11T21:21:27.1874433Z else
2023-01-11T21:21:27.1874714Z   TEST_COMMAND=.jenkins/pytorch/test.sh
2023-01-11T21:21:27.1874988Z fi
2023-01-11T21:21:27.1875191Z 
2023-01-11T21:21:27.1875513Z COMMIT_MESSAGES=$(git cherry -v "origin/${GIT_DEFAULT_BRANCH:-master}")
2023-01-11T21:21:27.1875835Z 
2023-01-11T21:21:27.1876116Z # sanitize the input commit message and PR body here:
2023-01-11T21:21:27.1876413Z #
2023-01-11T21:21:27.1876794Z # trim all new lines from commit messages + PR_BODY to avoid issues with batch environment
2023-01-11T21:21:27.1877290Z # variable copying. see https://github.com/pytorch/pytorch/pull/80043#issuecomment-1167796028
2023-01-11T21:21:27.1877721Z COMMIT_MESSAGES="${COMMIT_MESSAGES//[$'\n\r']}"
2023-01-11T21:21:27.1878040Z PR_BODY="${PR_BODY//[$'\n\r']}"
2023-01-11T21:21:27.1878296Z 
2023-01-11T21:21:27.1878635Z # then trim all special characters like single and double quotes to avoid unescaped inputs to
2023-01-11T21:21:27.1879013Z # wreak havoc internally
2023-01-11T21:21:27.1879336Z export COMMIT_MESSAGES="${COMMIT_MESSAGES//[\'\"]}"
2023-01-11T21:21:27.1879651Z export PR_BODY="${PR_BODY//[\'\"]}"
2023-01-11T21:21:27.1879913Z 
2023-01-11T21:21:27.1880227Z # detached container should get cleaned up by teardown_ec2_linux
2023-01-11T21:21:27.1880619Z # TODO: Stop building test binaries as part of the build phase
2023-01-11T21:21:27.1881001Z # Used for GPU_FLAG since that doesn't play nice
2023-01-11T21:21:27.1881329Z # shellcheck disable=SC2086,SC2090
2023-01-11T21:21:27.1881636Z container_name=$(docker run \
2023-01-11T21:21:27.1881896Z   ${GPU_FLAG:-} \
2023-01-11T21:21:27.1882167Z   -e BUILD_ENVIRONMENT \
2023-01-11T21:21:27.1882441Z   -e PR_NUMBER \
2023-01-11T21:21:27.1882692Z   -e GITHUB_ACTIONS \
2023-01-11T21:21:27.1882954Z   -e BASE_SHA \
2023-01-11T21:21:27.1883204Z   -e BRANCH \
2023-01-11T21:21:27.1883432Z   -e SHA1 \
2023-01-11T21:21:27.1883692Z   -e AWS_DEFAULT_REGION \
2023-01-11T21:21:27.1883964Z   -e IN_WHEEL_TEST \
2023-01-11T21:21:27.1884458Z   -e SHARD_NUMBER \
2023-01-11T21:21:27.1884736Z   -e TEST_CONFIG \
2023-01-11T21:21:27.1885003Z   -e NUM_TEST_SHARDS \
2023-01-11T21:21:27.1885269Z   -e PR_BODY \
2023-01-11T21:21:27.1885522Z   -e COMMIT_MESSAGES \
2023-01-11T21:21:27.1885808Z   -e CONTINUE_THROUGH_ERROR \
2023-01-11T21:21:27.1886109Z   -e PYTORCH_RETRY_TEST_CASES \
2023-01-11T21:21:27.1886416Z   -e PYTORCH_OVERRIDE_FLAKY_SIGNAL \
2023-01-11T21:21:27.1886705Z   -e PR_LABELS \
2023-01-11T21:21:27.1886994Z   -e MAX_JOBS="$(nproc --ignore=2)" \
2023-01-11T21:21:27.1887271Z   -e SCCACHE_BUCKET \
2023-01-11T21:21:27.1887551Z   -e SCCACHE_S3_KEY_PREFIX \
2023-01-11T21:21:27.1887822Z   -e XLA_CUDA \
2023-01-11T21:21:27.1888093Z   -e XLA_CLANG_CACHE_S3_BUCKET_NAME \
2023-01-11T21:21:27.1888415Z   -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK \
2023-01-11T21:21:27.1888746Z   -e PYTORCH_TEST_RERUN_DISABLED_TESTS \
2023-01-11T21:21:27.1889101Z   --env-file="/tmp/github_env_${GITHUB_RUN_ID}" \
2023-01-11T21:21:27.1889410Z   --ulimit stack=10485760:83886080 \
2023-01-11T21:21:27.1889825Z   --security-opt seccomp=unconfined \
2023-01-11T21:21:27.1890147Z   --cap-add=SYS_PTRACE \
2023-01-11T21:21:27.1890402Z   --ipc=host \
2023-01-11T21:21:27.1890741Z   --shm-size="${SHM_SIZE}" \
2023-01-11T21:21:27.1891002Z   --tty \
2023-01-11T21:21:27.1891228Z   --detach \
2023-01-11T21:21:27.1891501Z   --name="${container_name}" \
2023-01-11T21:21:27.1891779Z   --user jenkins \
2023-01-11T21:21:27.1892087Z   -v "${GITHUB_WORKSPACE}:/var/lib/jenkins/workspace" \
2023-01-11T21:21:27.1892434Z   -w /var/lib/jenkins/workspace \
2023-01-11T21:21:27.1892720Z   "${DOCKER_IMAGE}"
2023-01-11T21:21:27.1892963Z )
2023-01-11T21:21:27.1893250Z echo "DOCKER_CONTAINER_ID=${container_name}" >> "${GITHUB_ENV}"
2023-01-11T21:21:27.1893704Z docker exec -t "${container_name}" sh -c "pip install $(echo dist/*.whl)[opt-einsum] && ${TEST_COMMAND}"
2023-01-11T21:21:27.1905325Z shell: /usr/bin/bash -e {0}
2023-01-11T21:21:27.1905561Z env:
2023-01-11T21:21:27.1905798Z   GIT_DEFAULT_BRANCH: master
2023-01-11T21:21:27.1906069Z   GPU_FLAG: --gpus all
2023-01-11T21:21:27.1906387Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.7-py3.10-gcc7
2023-01-11T21:21:27.1906702Z   PR_NUMBER: 
2023-01-11T21:21:27.1906932Z   BRANCH: 
2023-01-11T21:21:27.1907194Z   SHA1: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:27.1907538Z   BASE_SHA: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:27.1907852Z   PYTORCH_RETRY_TEST_CASES: 1
2023-01-11T21:21:27.1908122Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2023-01-11T21:21:27.1908405Z   TEST_CONFIG: distributed
2023-01-11T21:21:27.1908659Z   SHARD_NUMBER: 1
2023-01-11T21:21:27.1908883Z   NUM_TEST_SHARDS: 3
2023-01-11T21:21:27.1909121Z   PR_BODY: 
2023-01-11T21:21:27.1909374Z   CONTINUE_THROUGH_ERROR: False
2023-01-11T21:21:27.1909711Z   SCCACHE_BUCKET: ossci-compiler-cache-circleci-v2
2023-01-11T21:21:27.1910024Z   SCCACHE_S3_KEY_PREFIX: trunk
2023-01-11T21:21:27.1910285Z   SHM_SIZE: 2g
2023-01-11T21:21:27.1910783Z   DOCKER_IMAGE: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:21:27.1911257Z   XLA_CUDA: 
2023-01-11T21:21:27.1911610Z   XLA_CLANG_CACHE_S3_BUCKET_NAME: ossci-compiler-clang-cache-circleci-xla
2023-01-11T21:21:27.1911994Z   PYTORCH_TEST_CUDA_MEM_LEAK_CHECK: 0
2023-01-11T21:21:27.1912279Z   PYTORCH_TEST_RERUN_DISABLED_TESTS: 0
2023-01-11T21:21:27.1912556Z ##[endgroup]
2023-01-11T21:21:27.1940978Z + [[ distributed == \m\u\l\t\i\g\p\u ]]
2023-01-11T21:21:27.1941471Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *onnx* ]]
2023-01-11T21:21:27.1941815Z + TEST_COMMAND=.jenkins/pytorch/test.sh
2023-01-11T21:21:27.1944954Z ++ git cherry -v origin/master
2023-01-11T21:21:27.2483768Z + COMMIT_MESSAGES='+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into '\''input'\''
2023-01-11T21:21:27.2484488Z + 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch
2023-01-11T21:21:27.2485087Z + 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e '\''other'\'' instead of '\''output'\'' in documentation'
2023-01-11T21:21:27.2486296Z + COMMIT_MESSAGES='+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into '\''input'\''+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e '\''other'\'' instead of '\''output'\'' in documentation'
2023-01-11T21:21:27.2486906Z + PR_BODY=
2023-01-11T21:21:27.2488586Z + export 'COMMIT_MESSAGES=+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation'
2023-01-11T21:21:27.2489931Z + COMMIT_MESSAGES='+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation'
2023-01-11T21:21:27.2490512Z + export PR_BODY=
2023-01-11T21:21:27.2490799Z + PR_BODY=
2023-01-11T21:21:27.2498192Z +++ nproc --ignore=2
2023-01-11T21:21:27.2510758Z ++ docker run --gpus all -e BUILD_ENVIRONMENT -e PR_NUMBER -e GITHUB_ACTIONS -e BASE_SHA -e BRANCH -e SHA1 -e AWS_DEFAULT_REGION -e IN_WHEEL_TEST -e SHARD_NUMBER -e TEST_CONFIG -e NUM_TEST_SHARDS -e PR_BODY -e COMMIT_MESSAGES -e CONTINUE_THROUGH_ERROR -e PYTORCH_RETRY_TEST_CASES -e PYTORCH_OVERRIDE_FLAKY_SIGNAL -e PR_LABELS -e MAX_JOBS=30 -e SCCACHE_BUCKET -e SCCACHE_S3_KEY_PREFIX -e XLA_CUDA -e XLA_CLANG_CACHE_S3_BUCKET_NAME -e PYTORCH_TEST_CUDA_MEM_LEAK_CHECK -e PYTORCH_TEST_RERUN_DISABLED_TESTS --env-file=/tmp/github_env_3896346758 --ulimit stack=10485760:83886080 --security-opt seccomp=unconfined --cap-add=SYS_PTRACE --ipc=host --shm-size=2g --tty --detach --name= --user jenkins -v /home/ec2-user/actions-runner/_work/pytorch/pytorch:/var/lib/jenkins/workspace -w /var/lib/jenkins/workspace 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T21:21:41.4496517Z + container_name=c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56
2023-01-11T21:21:41.4497042Z + echo DOCKER_CONTAINER_ID=c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56
2023-01-11T21:21:41.4502324Z ++ echo dist/torch-2.0.0a0+git8419ddd-cp310-cp310-linux_x86_64.whl
2023-01-11T21:21:41.4503996Z + docker exec -t c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56 sh -c 'pip install dist/torch-2.0.0a0+git8419ddd-cp310-cp310-linux_x86_64.whl[opt-einsum] && .jenkins/pytorch/test.sh'
2023-01-11T21:21:42.0144061Z Processing ./dist/torch-2.0.0a0+git8419ddd-cp310-cp310-linux_x86_64.whl
2023-01-11T21:21:42.9782839Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch==2.0.0a0+git8419ddd) (1.11.1)
2023-01-11T21:21:42.9786285Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch==2.0.0a0+git8419ddd) (4.4.0)
2023-01-11T21:21:42.9790840Z Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch==2.0.0a0+git8419ddd) (2.6.3)
2023-01-11T21:21:42.9807921Z Requirement already satisfied: opt-einsum>=3.3 in /opt/conda/lib/python3.10/site-packages (from torch==2.0.0a0+git8419ddd) (3.3.0)
2023-01-11T21:21:42.9888493Z Requirement already satisfied: numpy>=1.7 in /opt/conda/lib/python3.10/site-packages (from opt-einsum>=3.3->torch==2.0.0a0+git8419ddd) (1.21.2)
2023-01-11T21:21:43.0107468Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch==2.0.0a0+git8419ddd) (1.2.1)
2023-01-11T21:21:43.9355075Z Installing collected packages: torch
2023-01-11T21:21:53.5554588Z Successfully installed torch-2.0.0a0+git8419ddd
2023-01-11T21:21:53.7247226Z + echo 'Environment variables:'
2023-01-11T21:21:53.7247561Z Environment variables:
2023-01-11T21:21:53.7250712Z + env
2023-01-11T21:21:53.7256261Z SHARD_NUMBER=1
2023-01-11T21:21:53.7257125Z NV_LIBCUBLAS_DEV_VERSION=11.10.1.25-1
2023-01-11T21:21:53.7257528Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-7
2023-01-11T21:21:53.7258843Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-01-11T21:21:53.7260469Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.13.4-1+cuda11.7
2023-01-11T21:21:53.7260996Z UCC_HOME=/usr
2023-01-11T21:21:53.7261485Z BUILD_ENVIRONMENT=linux-bionic-cuda11.7-py3.10-gcc7
2023-01-11T21:21:53.7261836Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2023-01-11T21:21:53.7262228Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-7=11.7.3.21-1
2023-01-11T21:21:53.7262529Z INSTALLED_DB=yes
2023-01-11T21:21:53.7262766Z HOSTNAME=c3943a31ca1f
2023-01-11T21:21:53.7263043Z GITHUB_REF_NAME=ciflow/trunk/91627
2023-01-11T21:21:53.7263538Z GITHUB_API_URL=https://api.github.com
2023-01-11T21:21:53.7264036Z GITHUB_REPOSITORY_OWNER_ID=21003710
2023-01-11T21:21:53.7264562Z OPENSSL_DIR=/opt/openssl
2023-01-11T21:21:53.7264882Z UCC_COMMIT=1c7a7127186e7836f73aafbd7697bbc274a77eee
2023-01-11T21:21:53.7265843Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_c47620f0-df41-477f-a419-9e912b04118a
2023-01-11T21:21:53.7266805Z CUDA_PATH=/usr/local/cuda
2023-01-11T21:21:53.7267768Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2023-01-11T21:21:53.7268324Z GITHUB_RUN_ATTEMPT=1
2023-01-11T21:21:53.7268796Z TEST_CONFIG=distributed
2023-01-11T21:21:53.7269503Z NV_LIBNPP_VERSION=11.7.3.21-1
2023-01-11T21:21:53.7270268Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-7=11.7.50-1
2023-01-11T21:21:53.7270592Z GITHUB_REPOSITORY_OWNER=pytorch
2023-01-11T21:21:53.7270874Z GITHUB_ACTIONS=true
2023-01-11T21:21:53.7271136Z NVIDIA_VISIBLE_DEVICES=all
2023-01-11T21:21:53.7271424Z NV_NVPROF_VERSION=11.7.50-1
2023-01-11T21:21:53.7271740Z NV_LIBCUSPARSE_VERSION=11.7.3.50-1
2023-01-11T21:21:53.7272151Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/91627
2023-01-11T21:21:53.7279218Z NVIDIA_PRODUCT_NAME=CUDA
2023-01-11T21:21:53.7279808Z CI=true
2023-01-11T21:21:53.7280584Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2023-01-11T21:21:53.7281606Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-7=11.10.1.25-1
2023-01-11T21:21:53.7282223Z BRANCH=
2023-01-11T21:21:53.7282819Z GITHUB_HEAD_REF=
2023-01-11T21:21:53.7283601Z UCX_COMMIT=31e74cac7bee0ef66bef2af72e7d86d9c282e5ab
2023-01-11T21:21:53.7284026Z GITHUB_ACTOR=pytorch-bot[bot]
2023-01-11T21:21:53.7284987Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2023-01-11T21:21:53.7285480Z GITHUB_ACTION_REF=
2023-01-11T21:21:53.7285891Z NCCL_VERSION=2.13.4-1
2023-01-11T21:21:53.7286163Z GITHUB_ACTION=__self
2023-01-11T21:21:53.7286494Z GITHUB_REF_PROTECTED=false
2023-01-11T21:21:53.7287026Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2023-01-11T21:21:53.7287424Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2023-01-11T21:21:53.7289879Z ***
2023-01-11T21:21:53.7290206Z INSTALLED_VISION=yes
2023-01-11T21:21:53.7290474Z NVARCH=x86_64
2023-01-11T21:21:53.7290873Z NV_LIBCUSPARSE_DEV_VERSION=11.7.3.50-1
2023-01-11T21:21:53.7291231Z HOME=/var/lib/jenkins
2023-01-11T21:21:53.7291866Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_c47620f0-df41-477f-a419-9e912b04118a
2023-01-11T21:21:53.7292295Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2023-01-11T21:21:53.7292645Z NVIDIA_CUDA_END_OF_LIFE=1
2023-01-11T21:21:53.7292999Z GITHUB_ACTION_REPOSITORY=
2023-01-11T21:21:53.7293325Z GITHUB_REF_TYPE=tag
2023-01-11T21:21:53.7293717Z NV_LIBNCCL_PACKAGE_VERSION=2.13.4-1
2023-01-11T21:21:53.7294077Z GITHUB_RETENTION_DAYS=90
2023-01-11T21:21:53.7294479Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2023-01-11T21:21:53.7294969Z NV_LIBNCCL_PACKAGE=libnccl2=2.13.4-1+cuda11.7
2023-01-11T21:21:53.7295639Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_c47620f0-df41-477f-a419-9e912b04118a
2023-01-11T21:21:53.7296075Z DEBIAN_FRONTEND=noninteractive
2023-01-11T21:21:53.7296534Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2023-01-11T21:21:53.7296931Z GITHUB_REF=refs/tags/ciflow/trunk/91627
2023-01-11T21:21:53.7297285Z NV_CUDA_LIB_VERSION=11.7.0-1
2023-01-11T21:21:53.7297728Z GITHUB_SHA=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:53.7298109Z INSTALLED_PROTOBUF=yes
2023-01-11T21:21:53.7298470Z GITHUB_REPOSITORY_ID=65600975
2023-01-11T21:21:53.7298762Z GITHUB_RUN_ID=3896346758
2023-01-11T21:21:53.7299218Z NV_LIBNPP_PACKAGE=libnpp-11-7=11.7.3.21-1
2023-01-11T21:21:53.7299908Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2023-01-11T21:21:53.7300266Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2023-01-11T21:21:53.7300675Z NV_NVTX_VERSION=11.7.50-1
2023-01-11T21:21:53.7301011Z CONTINUE_THROUGH_ERROR=False
2023-01-11T21:21:53.7301377Z GITHUB_SERVER_URL=https://github.com
2023-01-11T21:21:53.7301736Z MAX_JOBS=30
2023-01-11T21:21:53.7302048Z GITHUB_ACTOR_ID=54816060
2023-01-11T21:21:53.7302532Z NV_LIBCUBLAS_VERSION=11.10.1.25-1
2023-01-11T21:21:53.7303012Z NV_LIBCUBLAS_PACKAGE=libcublas-11-7=11.10.1.25-1
2023-01-11T21:21:53.7303632Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2023-01-11T21:21:53.7304128Z UCX_HOME=/usr
2023-01-11T21:21:53.7304437Z PYTORCH_RETRY_TEST_CASES=1
2023-01-11T21:21:53.7304833Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2023-01-11T21:21:53.7305227Z BASE_SHA=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:53.7305687Z NV_CUDA_CUDART_DEV_VERSION=11.7.60-1
2023-01-11T21:21:53.7306011Z PR_BODY=
2023-01-11T21:21:53.7306258Z GITHUB_BASE_REF=
2023-01-11T21:21:53.7306556Z TERM=xterm
2023-01-11T21:21:53.7306866Z XLA_CUDA=
2023-01-11T21:21:53.7307157Z NV_NVML_DEV_VERSION=11.7.50-1
2023-01-11T21:21:53.7307497Z TORCH_CUDA_ARCH_LIST=Maxwell
2023-01-11T21:21:53.7354415Z CUDA_VERSION=11.7.0
2023-01-11T21:21:53.7354805Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-7
2023-01-11T21:21:53.7355119Z OPENSSL_ROOT_DIR=/opt/openssl
2023-01-11T21:21:53.7355683Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_c47620f0-df41-477f-a419-9e912b04118a
2023-01-11T21:21:53.7356071Z GITHUB_JOB=test
2023-01-11T21:21:53.7356306Z SCCACHE_S3_KEY_PREFIX=trunk
2023-01-11T21:21:53.7356897Z COMMIT_MESSAGES=+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation
2023-01-11T21:21:53.7357564Z NVIDIA_DRIVER_CAPABILITIES=compute,utility
2023-01-11T21:21:53.7357858Z NUM_TEST_SHARDS=3
2023-01-11T21:21:53.7358085Z PR_NUMBER=
2023-01-11T21:21:53.7358594Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_c47620f0-df41-477f-a419-9e912b04118a
2023-01-11T21:21:53.7358949Z SHLVL=1
2023-01-11T21:21:53.7359269Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-7
2023-01-11T21:21:53.7359597Z GITHUB_REPOSITORY=pytorch/pytorch
2023-01-11T21:21:53.7360777Z NVIDIA_REQUIRE_CUDA=cuda>=11.7 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=510,driver<511 brand=unknown,driver>=510,driver<511 brand=nvidia,driver>=510,driver<511 brand=nvidiartx,driver>=510,driver<511 brand=quadro,driver>=510,driver<511 brand=quadrortx,driver>=510,driver<511 brand=titan,driver>=510,driver<511 brand=titanrtx,driver>=510,driver<511 brand=geforce,driver>=510,driver<511 brand=geforcertx,driver>=510,driver<511
2023-01-11T21:21:53.7361905Z NV_LIBNPP_DEV_VERSION=11.7.3.21-1
2023-01-11T21:21:53.7362219Z SHA1=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:53.7362519Z GITHUB_EVENT_NAME=push
2023-01-11T21:21:53.7362805Z NV_CUDA_CUDART_VERSION=11.7.60-1
2023-01-11T21:21:53.7363160Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2023-01-11T21:21:53.7363456Z GITHUB_RUN_NUMBER=22986
2023-01-11T21:21:53.7363704Z GITHUB_WORKFLOW=trunk
2023-01-11T21:21:53.7364125Z PATH=/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T21:21:53.7365143Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.13.4-1
2023-01-11T21:21:53.7365486Z GITHUB_WORKFLOW_SHA=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:53.7365969Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T21:21:53.7366396Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot]
2023-01-11T21:21:53.7366681Z _=/usr/bin/env
2023-01-11T21:21:53.7367047Z ++ python -c 'import site; print(site.getsitepackages()[0])'
2023-01-11T21:21:53.7480320Z + TORCH_INSTALL_DIR=/opt/conda/lib/python3.10/site-packages/torch
2023-01-11T21:21:53.7481593Z + TORCH_BIN_DIR=/opt/conda/lib/python3.10/site-packages/torch/bin
2023-01-11T21:21:53.7482153Z + TORCH_LIB_DIR=/opt/conda/lib/python3.10/site-packages/torch/lib
2023-01-11T21:21:53.7482625Z + TORCH_TEST_DIR=/opt/conda/lib/python3.10/site-packages/torch/test
2023-01-11T21:21:53.7483063Z + BUILD_DIR=build
2023-01-11T21:21:53.7483339Z + BUILD_RENAMED_DIR=build_renamed
2023-01-11T21:21:53.7483597Z + BUILD_BIN_DIR=build/bin
2023-01-11T21:21:53.7483864Z + export VALGRIND=ON
2023-01-11T21:21:53.7484115Z + VALGRIND=ON
2023-01-11T21:21:53.7484851Z + export TORCH_INDUCTOR_INSTALL_GXX=ON
2023-01-11T21:21:53.7485158Z + TORCH_INDUCTOR_INSTALL_GXX=ON
2023-01-11T21:21:53.7485592Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *clang9* ]]
2023-01-11T21:21:53.7486016Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 != *bazel* ]]
2023-01-11T21:21:53.7488436Z ++ realpath build/custom_test_artifacts
2023-01-11T21:21:53.7495466Z + CUSTOM_TEST_ARTIFACT_BUILD_DIR=/var/lib/jenkins/workspace/build/custom_test_artifacts
2023-01-11T21:21:53.7499551Z ++ dirname .jenkins/pytorch/test.sh
2023-01-11T21:21:53.7506197Z + source .jenkins/pytorch/common.sh
2023-01-11T21:21:53.7510516Z +++ dirname .jenkins/pytorch/common.sh
2023-01-11T21:21:53.7521086Z ++ source .jenkins/pytorch/common_utils.sh
2023-01-11T21:21:53.7523427Z +++ declare -f -t trap_add
2023-01-11T21:21:53.7531366Z ++ set -ex
2023-01-11T21:21:53.7531848Z ++ [[ linux-bionic-cuda11.7-py3.10-gcc7 == *rocm* ]]
2023-01-11T21:21:53.7532312Z ++ BUILD_TEST_LIBTORCH=0
2023-01-11T21:21:53.7532651Z + echo 'Environment variables'
2023-01-11T21:21:53.7532915Z Environment variables
2023-01-11T21:21:53.7533143Z + env
2023-01-11T21:21:53.7540062Z SHARD_NUMBER=1
2023-01-11T21:21:53.7540611Z NV_LIBCUBLAS_DEV_VERSION=11.10.1.25-1
2023-01-11T21:21:53.7541008Z NV_CUDA_COMPAT_PACKAGE=cuda-compat-11-7
2023-01-11T21:21:53.7541348Z LD_LIBRARY_PATH=/usr/local/nvidia/lib:/usr/local/nvidia/lib64
2023-01-11T21:21:53.7541789Z NV_LIBNCCL_DEV_PACKAGE=libnccl-dev=2.13.4-1+cuda11.7
2023-01-11T21:21:53.7542082Z UCC_HOME=/usr
2023-01-11T21:21:53.7542680Z BUILD_ENVIRONMENT=linux-bionic-cuda11.7-py3.10-gcc7
2023-01-11T21:21:53.7543042Z PYTORCH_TEST_CUDA_MEM_LEAK_CHECK=0
2023-01-11T21:21:53.7543432Z NV_LIBNPP_DEV_PACKAGE=libnpp-dev-11-7=11.7.3.21-1
2023-01-11T21:21:53.7543740Z INSTALLED_DB=yes
2023-01-11T21:21:53.7543973Z HOSTNAME=c3943a31ca1f
2023-01-11T21:21:53.7544250Z GITHUB_REF_NAME=ciflow/trunk/91627
2023-01-11T21:21:53.7544573Z GITHUB_API_URL=https://api.github.com
2023-01-11T21:21:53.7544873Z GITHUB_REPOSITORY_OWNER_ID=21003710
2023-01-11T21:21:53.7545160Z OPENSSL_DIR=/opt/openssl
2023-01-11T21:21:53.7545519Z UCC_COMMIT=1c7a7127186e7836f73aafbd7697bbc274a77eee
2023-01-11T21:21:53.7546518Z GITHUB_STEP_SUMMARY=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/step_summary_c47620f0-df41-477f-a419-9e912b04118a
2023-01-11T21:21:53.7547222Z CUDA_PATH=/usr/local/cuda
2023-01-11T21:21:53.7547727Z GITHUB_ACTION_PATH=/home/ec2-user/actions-runner/_work/pytorch/pytorch/./.github/actions/setup-linux
2023-01-11T21:21:53.7548380Z GITHUB_RUN_ATTEMPT=1
2023-01-11T21:21:53.7548853Z TEST_CONFIG=distributed
2023-01-11T21:21:53.7549402Z NV_LIBNPP_VERSION=11.7.3.21-1
2023-01-11T21:21:53.7549949Z NV_NVPROF_DEV_PACKAGE=cuda-nvprof-11-7=11.7.50-1
2023-01-11T21:21:53.7550488Z GITHUB_REPOSITORY_OWNER=pytorch
2023-01-11T21:21:53.7550760Z GITHUB_ACTIONS=true
2023-01-11T21:21:53.7551025Z NVIDIA_VISIBLE_DEVICES=all
2023-01-11T21:21:53.7551315Z NV_NVPROF_VERSION=11.7.50-1
2023-01-11T21:21:53.7551633Z NV_LIBCUSPARSE_VERSION=11.7.3.50-1
2023-01-11T21:21:53.7552018Z GITHUB_WORKFLOW_REF=pytorch/pytorch/.github/workflows/trunk.yml@refs/tags/ciflow/trunk/91627
2023-01-11T21:21:53.7552371Z NVIDIA_PRODUCT_NAME=CUDA
2023-01-11T21:21:53.7552619Z CI=true
2023-01-11T21:21:53.7552873Z PYTORCH_OVERRIDE_FLAKY_SIGNAL=1
2023-01-11T21:21:53.7553263Z NV_LIBCUBLAS_DEV_PACKAGE=libcublas-dev-11-7=11.10.1.25-1
2023-01-11T21:21:53.7553557Z BRANCH=
2023-01-11T21:21:53.7553833Z GITHUB_HEAD_REF=
2023-01-11T21:21:53.7554146Z UCX_COMMIT=31e74cac7bee0ef66bef2af72e7d86d9c282e5ab
2023-01-11T21:21:53.7554638Z GITHUB_ACTOR=pytorch-bot[bot]
2023-01-11T21:21:53.7554978Z CMAKE_CUDA_COMPILER_LAUNCHER=/opt/cache/bin/sccache
2023-01-11T21:21:53.7555283Z GITHUB_ACTION_REF=
2023-01-11T21:21:53.7555546Z NCCL_VERSION=2.13.4-1
2023-01-11T21:21:53.7555876Z GITHUB_ACTION=__self
2023-01-11T21:21:53.7556124Z VALGRIND=ON
2023-01-11T21:21:53.7556361Z GITHUB_REF_PROTECTED=false
2023-01-11T21:21:53.7556822Z XLA_CLANG_CACHE_S3_BUCKET_NAME=ossci-compiler-clang-cache-circleci-xla
2023-01-11T21:21:53.7557211Z PYTORCH_TEST_RERUN_DISABLED_TESTS=0
2023-01-11T21:21:53.7557558Z ***
2023-01-11T21:21:53.7557792Z INSTALLED_VISION=yes
2023-01-11T21:21:53.7558044Z NVARCH=x86_64
2023-01-11T21:21:53.7558331Z NV_LIBCUSPARSE_DEV_VERSION=11.7.3.50-1
2023-01-11T21:21:53.7558612Z HOME=/var/lib/jenkins
2023-01-11T21:21:53.7559145Z GITHUB_STATE=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/save_state_c47620f0-df41-477f-a419-9e912b04118a
2023-01-11T21:21:53.7559557Z CARGO_NET_GIT_FETCH_WITH_CLI=true
2023-01-11T21:21:53.7559818Z NVIDIA_CUDA_END_OF_LIFE=1
2023-01-11T21:21:53.7560099Z GITHUB_ACTION_REPOSITORY=
2023-01-11T21:21:53.7560359Z GITHUB_REF_TYPE=tag
2023-01-11T21:21:53.7560643Z NV_LIBNCCL_PACKAGE_VERSION=2.13.4-1
2023-01-11T21:21:53.7560935Z GITHUB_RETENTION_DAYS=90
2023-01-11T21:21:53.7561322Z SCCACHE_BUCKET=ossci-compiler-cache-circleci-v2
2023-01-11T21:21:53.7561721Z NV_LIBNCCL_PACKAGE=libnccl2=2.13.4-1+cuda11.7
2023-01-11T21:21:53.7562271Z GITHUB_ENV=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_env_c47620f0-df41-477f-a419-9e912b04118a
2023-01-11T21:21:53.7562676Z DEBIAN_FRONTEND=noninteractive
2023-01-11T21:21:53.7563011Z NV_LIBNCCL_DEV_PACKAGE_NAME=libnccl-dev
2023-01-11T21:21:53.7563324Z GITHUB_REF=refs/tags/ciflow/trunk/91627
2023-01-11T21:21:53.7563653Z NV_CUDA_LIB_VERSION=11.7.0-1
2023-01-11T21:21:53.7563977Z GITHUB_SHA=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:53.7564595Z INSTALLED_PROTOBUF=yes
2023-01-11T21:21:53.7564892Z GITHUB_REPOSITORY_ID=65600975
2023-01-11T21:21:53.7565171Z GITHUB_RUN_ID=3896346758
2023-01-11T21:21:53.7565531Z NV_LIBNPP_PACKAGE=libnpp-11-7=11.7.3.21-1
2023-01-11T21:21:53.7565862Z NV_LIBNCCL_PACKAGE_NAME=libnccl2
2023-01-11T21:21:53.7566182Z LIBRARY_PATH=/usr/local/cuda/lib64/stubs
2023-01-11T21:21:53.7566499Z NV_NVTX_VERSION=11.7.50-1
2023-01-11T21:21:53.7566786Z CONTINUE_THROUGH_ERROR=False
2023-01-11T21:21:53.7567113Z GITHUB_SERVER_URL=https://github.com
2023-01-11T21:21:53.7567385Z MAX_JOBS=30
2023-01-11T21:21:53.7567644Z GITHUB_ACTOR_ID=54816060
2023-01-11T21:21:53.7567966Z NV_LIBCUBLAS_VERSION=11.10.1.25-1
2023-01-11T21:21:53.7568351Z NV_LIBCUBLAS_PACKAGE=libcublas-11-7=11.10.1.25-1
2023-01-11T21:21:53.7568874Z GITHUB_EVENT_PATH=/home/ec2-user/actions-runner/_work/_temp/_github_workflow/event.json
2023-01-11T21:21:53.7569247Z UCX_HOME=/usr
2023-01-11T21:21:53.7569498Z PYTORCH_RETRY_TEST_CASES=1
2023-01-11T21:21:53.7569854Z GITHUB_GRAPHQL_URL=https://api.github.com/graphql
2023-01-11T21:21:53.7570237Z BASE_SHA=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:53.7570586Z NV_CUDA_CUDART_DEV_VERSION=11.7.60-1
2023-01-11T21:21:53.7570843Z PR_BODY=
2023-01-11T21:21:53.7571062Z GITHUB_BASE_REF=
2023-01-11T21:21:53.7571293Z TERM=xterm
2023-01-11T21:21:53.7571557Z TORCH_INDUCTOR_INSTALL_GXX=ON
2023-01-11T21:21:53.7571818Z XLA_CUDA=
2023-01-11T21:21:53.7572088Z NV_NVML_DEV_VERSION=11.7.50-1
2023-01-11T21:21:53.7572376Z TORCH_CUDA_ARCH_LIST=Maxwell
2023-01-11T21:21:53.7572651Z CUDA_VERSION=11.7.0
2023-01-11T21:21:53.7572995Z NV_LIBCUBLAS_PACKAGE_NAME=libcublas-11-7
2023-01-11T21:21:53.7573317Z OPENSSL_ROOT_DIR=/opt/openssl
2023-01-11T21:21:53.7573890Z GITHUB_PATH=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/add_path_c47620f0-df41-477f-a419-9e912b04118a
2023-01-11T21:21:53.7574300Z GITHUB_JOB=test
2023-01-11T21:21:53.7574552Z SCCACHE_S3_KEY_PREFIX=trunk
2023-01-11T21:21:53.7575293Z COMMIT_MESSAGES=+ 52a16ce42647731c772e14e7175afa40fda07b3d make torchgen rename also Number arguments into input+ 87db01a53ecb702267ec36787654e418a52f8e93 fix torch.where signature mismatch+ 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e other instead of output in documentation
2023-01-11T21:21:53.7575957Z NVIDIA_DRIVER_CAPABILITIES=compute,utility
2023-01-11T21:21:53.7576321Z NUM_TEST_SHARDS=3
2023-01-11T21:21:53.7576573Z PR_NUMBER=
2023-01-11T21:21:53.7577135Z GITHUB_OUTPUT=/home/ec2-user/actions-runner/_work/_temp/_runner_file_commands/set_output_c47620f0-df41-477f-a419-9e912b04118a
2023-01-11T21:21:53.7577523Z SHLVL=1
2023-01-11T21:21:53.7577893Z NV_LIBCUBLAS_DEV_PACKAGE_NAME=libcublas-dev-11-7
2023-01-11T21:21:53.7578244Z GITHUB_REPOSITORY=pytorch/pytorch
2023-01-11T21:21:53.7579525Z NVIDIA_REQUIRE_CUDA=cuda>=11.7 brand=tesla,driver>=450,driver<451 brand=tesla,driver>=470,driver<471 brand=unknown,driver>=470,driver<471 brand=nvidia,driver>=470,driver<471 brand=nvidiartx,driver>=470,driver<471 brand=geforce,driver>=470,driver<471 brand=geforcertx,driver>=470,driver<471 brand=quadro,driver>=470,driver<471 brand=quadrortx,driver>=470,driver<471 brand=titan,driver>=470,driver<471 brand=titanrtx,driver>=470,driver<471 brand=tesla,driver>=510,driver<511 brand=unknown,driver>=510,driver<511 brand=nvidia,driver>=510,driver<511 brand=nvidiartx,driver>=510,driver<511 brand=quadro,driver>=510,driver<511 brand=quadrortx,driver>=510,driver<511 brand=titan,driver>=510,driver<511 brand=titanrtx,driver>=510,driver<511 brand=geforce,driver>=510,driver<511 brand=geforcertx,driver>=510,driver<511
2023-01-11T21:21:53.7580738Z NV_LIBNPP_DEV_VERSION=11.7.3.21-1
2023-01-11T21:21:53.7581042Z SHA1=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:53.7581333Z GITHUB_EVENT_NAME=push
2023-01-11T21:21:53.7581691Z NV_CUDA_CUDART_VERSION=11.7.60-1
2023-01-11T21:21:53.7582069Z TORCH_NVCC_FLAGS=-Xfatbin -compress-all
2023-01-11T21:21:53.7582358Z GITHUB_RUN_NUMBER=22986
2023-01-11T21:21:53.7582619Z GITHUB_WORKFLOW=trunk
2023-01-11T21:21:53.7583061Z PATH=/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T21:21:53.7583536Z NV_LIBNCCL_DEV_PACKAGE_VERSION=2.13.4-1
2023-01-11T21:21:53.7583910Z GITHUB_WORKFLOW_SHA=8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T21:21:53.7584409Z GITHUB_WORKSPACE=/home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T21:21:53.7584847Z GITHUB_TRIGGERING_ACTOR=pytorch-bot[bot]
2023-01-11T21:21:53.7585139Z _=/usr/bin/env
2023-01-11T21:21:53.7585440Z + echo 'Testing pytorch'
2023-01-11T21:21:53.7585693Z Testing pytorch
2023-01-11T21:21:53.7585978Z + export LANG=C.UTF-8
2023-01-11T21:21:53.7586258Z + LANG=C.UTF-8
2023-01-11T21:21:53.7586481Z + PR_NUMBER=
2023-01-11T21:21:53.7586733Z + [[ distributed == \d\e\f\a\u\l\t ]]
2023-01-11T21:21:53.7587040Z + [[ distributed == \d\i\s\t\r\i\b\u\t\e\d ]]
2023-01-11T21:21:53.7587441Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *rocm* ]]
2023-01-11T21:21:53.7587759Z + [[ distributed == \s\l\o\w ]]
2023-01-11T21:21:53.7588198Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *slow-gradcheck* ]]
2023-01-11T21:21:53.7588663Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *cuda* ]]
2023-01-11T21:21:53.7589019Z + export PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2023-01-11T21:21:53.7589359Z + PYTORCH_TESTING_DEVICE_ONLY_FOR=cuda
2023-01-11T21:21:53.7589666Z + [[ distributed == *crossref* ]]
2023-01-11T21:21:53.7589943Z + [[ distributed == *dynamo* ]]
2023-01-11T21:21:53.7590236Z + [[ distributed == *inductor* ]]
2023-01-11T21:21:53.7590647Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *rocm* ]]
2023-01-11T21:21:53.7591088Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 != *-bazel-* ]]
2023-01-11T21:21:53.7591493Z + pip_install --user ninja==1.10.2
2023-01-11T21:21:53.7591917Z + pip install --progress-bar off --user ninja==1.10.2
2023-01-11T21:21:54.2939972Z Collecting ninja==1.10.2
2023-01-11T21:21:54.3178191Z   Downloading ninja-1.10.2-py2.py3-none-manylinux_2_5_x86_64.manylinux1_x86_64.whl (108 kB)
2023-01-11T21:21:55.2072181Z Installing collected packages: ninja
2023-01-11T21:21:55.2173188Z   WARNING: The script ninja is installed in '/var/lib/jenkins/.local/bin' which is not on PATH.
2023-01-11T21:21:55.2174058Z   Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
2023-01-11T21:21:55.2221694Z Successfully installed ninja-1.10.2
2023-01-11T21:21:55.2875419Z + export PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T21:21:55.2876065Z + PATH=/var/lib/jenkins/.local/bin:/opt/cache/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
2023-01-11T21:21:55.2877996Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *asan* ]]
2023-01-11T21:21:55.2878553Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *-tsan* ]]
2023-01-11T21:21:55.2878915Z + [[ distributed == \n\o\g\p\u\_\N\O\_\A\V\X\2 ]]
2023-01-11T21:21:55.2879224Z + [[ distributed == \n\o\g\p\u\_\A\V\X\5\1\2 ]]
2023-01-11T21:21:55.2887328Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *tbb* ]]
2023-01-11T21:21:55.2902947Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *libtorch* ]]
2023-01-11T21:21:55.2903391Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *-bazel-* ]]
2023-01-11T21:21:55.2903834Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *-tsan* ]]
2023-01-11T21:21:55.2906592Z + cd test
2023-01-11T21:21:55.2906986Z + python -c 'import torch; print(torch.__config__.show())'
2023-01-11T21:21:56.9351739Z PyTorch built with:
2023-01-11T21:21:56.9352191Z   - GCC 7.5
2023-01-11T21:21:56.9352463Z   - C++ Version: 201703
2023-01-11T21:21:56.9353010Z   - Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2023-01-11T21:21:56.9353591Z   - Intel(R) MKL-DNN v2.7.2 (Git Hash fbec3e25a559ee252022ae066817b204e106a6ba)
2023-01-11T21:21:56.9354023Z   - OpenMP 201511 (a.k.a. OpenMP 4.5)
2023-01-11T21:21:56.9354388Z   - LAPACK is enabled (usually provided by MKL)
2023-01-11T21:21:56.9354729Z   - NNPACK is enabled
2023-01-11T21:21:56.9355045Z   - CPU capability usage: AVX2
2023-01-11T21:21:56.9355352Z   - CUDA Runtime 11.7
2023-01-11T21:21:56.9355747Z   - NVCC architecture flags: -gencode;arch=compute_52,code=sm_52
2023-01-11T21:21:56.9356094Z   - CuDNN 8.5
2023-01-11T21:21:56.9356356Z   - Magma 2.6.1
2023-01-11T21:21:56.9359500Z   - Build settings: BLAS_INFO=mkl, BUILD_TYPE=Release, CUDA_VERSION=11.7, CUDNN_VERSION=8.5.0, CXX_COMPILER=/opt/cache/bin/c++, CXX_FLAGS= -Wno-deprecated -fvisibility-inlines-hidden -DUSE_PTHREADPOOL -fopenmp -DNDEBUG -DUSE_KINETO -DLIBKINETO_NOROCTRACER -DUSE_FBGEMM -DUSE_QNNPACK -DUSE_PYTORCH_QNNPACK -DUSE_XNNPACK -DSYMBOLICATE_MOBILE_DEBUG_HANDLE -DEDGE_PROFILER_USE_KINETO -O2 -fPIC -Wall -Wextra -Werror=return-type -Werror=non-virtual-dtor -Wnarrowing -Wno-missing-field-initializers -Wno-type-limits -Wno-array-bounds -Wno-unknown-pragmas -Wunused-local-typedefs -Wno-unused-parameter -Wno-unused-function -Wno-unused-result -Wno-strict-overflow -Wno-strict-aliasing -Wno-error=deprecated-declarations -Wno-stringop-overflow -Wno-psabi -Wno-error=pedantic -Wno-error=redundant-decls -Wno-error=old-style-cast -fdiagnostics-color=always -faligned-new -Werror -Wno-unused-but-set-variable -Wno-maybe-uninitialized -fno-math-errno -fno-trapping-math -Werror=format -Wno-stringop-overflow, FORCE_FALLBACK_CUDA_MPI=1, LAPACK_INFO=mkl, PERF_WITH_AVX=1, PERF_WITH_AVX2=1, PERF_WITH_AVX512=1, TORCH_DISABLE_GPU_ASSERTS=ON, TORCH_VERSION=2.0.0, USE_CUDA=ON, USE_CUDNN=ON, USE_EXCEPTION_PTR=1, USE_GFLAGS=OFF, USE_GLOG=OFF, USE_MKL=ON, USE_MKLDNN=ON, USE_MPI=ON, USE_NCCL=ON, USE_NNPACK=ON, USE_OPENMP=ON, USE_ROCM=OFF, 
2023-01-11T21:21:56.9361758Z 
2023-01-11T21:21:57.1600402Z + cd test
2023-01-11T21:21:57.1600965Z + python -c 'import torch; print(torch.__config__.parallel_info())'
2023-01-11T21:21:58.7268743Z ATen/Parallel:
2023-01-11T21:21:58.7286031Z 	at::get_num_threads() : 16
2023-01-11T21:21:58.7286375Z 	at::get_num_interop_threads() : 16
2023-01-11T21:21:58.7286676Z OpenMP 201511 (a.k.a. OpenMP 4.5)
2023-01-11T21:21:58.7286940Z 	omp_get_max_threads() : 16
2023-01-11T21:21:58.7287870Z Intel(R) oneAPI Math Kernel Library Version 2022.0-Product Build 20211112 for Intel(R) 64 architecture applications
2023-01-11T21:21:58.7288315Z 	mkl_get_max_threads() : 16
2023-01-11T21:21:58.7288875Z Intel(R) MKL-DNN v2.7.2 (Git Hash fbec3e25a559ee252022ae066817b204e106a6ba)
2023-01-11T21:21:58.7289252Z std::thread::hardware_concurrency() : 32
2023-01-11T21:21:58.7289549Z Environment variables:
2023-01-11T21:21:58.7289806Z 	OMP_NUM_THREADS : [not set]
2023-01-11T21:21:58.7290078Z 	MKL_NUM_THREADS : [not set]
2023-01-11T21:21:58.7290360Z ATen parallel backend: OpenMP
2023-01-11T21:21:58.7290544Z 
2023-01-11T21:21:58.9518201Z + [[ distributed == *backward* ]]
2023-01-11T21:21:58.9518634Z + [[ distributed == *xla* ]]
2023-01-11T21:21:58.9518932Z + [[ distributed == \j\i\t\_\l\e\g\a\c\y ]]
2023-01-11T21:21:58.9519488Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *libtorch* ]]
2023-01-11T21:21:58.9519836Z + [[ distributed == distributed ]]
2023-01-11T21:21:58.9520111Z + install_filelock
2023-01-11T21:21:58.9520368Z + pip_install filelock
2023-01-11T21:21:58.9520726Z + pip install --progress-bar off filelock
2023-01-11T21:21:59.4704198Z Collecting filelock
2023-01-11T21:21:59.4933322Z   Downloading filelock-3.9.0-py3-none-any.whl (9.7 kB)
2023-01-11T21:22:00.3574326Z Installing collected packages: filelock
2023-01-11T21:22:00.3929496Z Successfully installed filelock-3.9.0
2023-01-11T21:22:00.4615102Z + install_triton
2023-01-11T21:22:00.4615386Z + local commit
2023-01-11T21:22:00.4615812Z + [[ distributed == *rocm* ]]
2023-01-11T21:22:00.4619602Z ++ get_pinned_commit triton
2023-01-11T21:22:00.4619913Z ++ cat .github/ci_commit_pins/triton.txt
2023-01-11T21:22:00.4634446Z + commit=0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T21:22:00.4635362Z + pip_install --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2023-01-11T21:22:00.4636110Z + pip install --progress-bar off --user git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2023-01-11T21:22:00.9272928Z Collecting git+https://github.com/openai/triton@0d7e7532279e45672555e344646f5c19c3972331#subdirectory=python
2023-01-11T21:22:00.9278188Z   Cloning https://github.com/openai/triton (to revision 0d7e7532279e45672555e344646f5c19c3972331) to /tmp/pip-req-build-f_ogbcqw
2023-01-11T21:22:00.9299386Z   Running command git clone --filter=blob:none --quiet https://github.com/openai/triton /tmp/pip-req-build-f_ogbcqw
2023-01-11T21:22:01.7880314Z   Running command git rev-parse -q --verify 'sha^0d7e7532279e45672555e344646f5c19c3972331'
2023-01-11T21:22:01.7902211Z   Running command git fetch -q https://github.com/openai/triton 0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T21:22:02.2223669Z   Running command git checkout -q 0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T21:22:02.6255004Z   Resolved https://github.com/openai/triton to commit 0d7e7532279e45672555e344646f5c19c3972331
2023-01-11T21:22:02.6256106Z   Running command git submodule update --init --recursive -q
2023-01-11T21:22:03.2532245Z   Preparing metadata (setup.py) ... [?25l- done
2023-01-11T21:22:03.4588965Z [?25hCollecting cmake
2023-01-11T21:22:03.4811848Z   Downloading cmake-3.25.0-py2.py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (23.7 MB)
2023-01-11T21:22:03.8563209Z Requirement already satisfied: filelock in /opt/conda/lib/python3.10/site-packages (from triton==2.0.0) (3.9.0)
2023-01-11T21:22:03.8566661Z Requirement already satisfied: torch in /opt/conda/lib/python3.10/site-packages (from triton==2.0.0) (2.0.0a0+git8419ddd)
2023-01-11T21:22:03.8822440Z Requirement already satisfied: networkx in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (2.6.3)
2023-01-11T21:22:03.8826805Z Requirement already satisfied: sympy in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (1.11.1)
2023-01-11T21:22:03.8831332Z Requirement already satisfied: typing-extensions in /opt/conda/lib/python3.10/site-packages (from torch->triton==2.0.0) (4.4.0)
2023-01-11T21:22:03.9044052Z Requirement already satisfied: mpmath>=0.19 in /opt/conda/lib/python3.10/site-packages (from sympy->torch->triton==2.0.0) (1.2.1)
2023-01-11T21:22:03.9115328Z Building wheels for collected packages: triton
2023-01-11T21:22:56.6078171Z   Building wheel for triton (setup.py) ... [?25l- \ | / - \ | / - \ | done
2023-01-11T21:22:56.6561976Z [?25h  Created wheel for triton: filename=triton-2.0.0-cp310-cp310-linux_x86_64.whl size=15377935 sha256=5da9406561ac8badedd5249cb6971ba144453b0bf86010b8b957cc1061a0fbbe
2023-01-11T21:22:56.6563403Z   Stored in directory: /var/lib/jenkins/.cache/pip/wheels/3f/1d/23/1c2bc47d618a44f9c949aea4b7e355e737a1f1ed208f009295
2023-01-11T21:22:56.6583749Z Successfully built triton
2023-01-11T21:22:57.5490643Z Installing collected packages: cmake, triton
2023-01-11T21:22:59.5342297Z Successfully installed cmake-3.25.0 triton-2.0.0
2023-01-11T21:22:59.6384618Z + pip_install --user jinja2
2023-01-11T21:22:59.6385073Z + pip install --progress-bar off --user jinja2
2023-01-11T21:23:00.6883204Z Collecting jinja2
2023-01-11T21:23:00.7092668Z   Downloading Jinja2-3.1.2-py3-none-any.whl (133 kB)
2023-01-11T21:23:00.9776113Z Requirement already satisfied: MarkupSafe>=2.0 in /opt/conda/lib/python3.10/site-packages (from jinja2) (2.1.1)
2023-01-11T21:23:01.8628168Z Installing collected packages: jinja2
2023-01-11T21:23:01.9662118Z Successfully installed jinja2-3.1.2
2023-01-11T21:23:02.0344254Z + test_distributed
2023-01-11T21:23:02.0344747Z + echo 'Testing distributed python tests'
2023-01-11T21:23:02.0345051Z Testing distributed python tests
2023-01-11T21:23:02.0345501Z + python test/run_test.py --distributed-tests --shard 1 3 --verbose
2023-01-11T21:23:04.2634214Z Ignoring disabled issues:  []
2023-01-11T21:23:04.3026700Z /var/lib/jenkins/workspace/test/run_test.py:1169: DeprecationWarning: distutils Version classes are deprecated. Use packaging.version instead.
2023-01-11T21:23:04.3027289Z   if torch.version.cuda is not None and LooseVersion(torch.version.cuda) >= "11.6":
2023-01-11T21:23:04.3034179Z Found test time stats from artifacts
2023-01-11T21:23:04.3053567Z Selected tests:
2023-01-11T21:23:04.3053902Z  distributed/algorithms/quantization/test_quantization
2023-01-11T21:23:04.3054262Z  distributed/test_distributed_spawn
2023-01-11T21:23:04.3056363Z  distributed/rpc/test_tensorpipe_agent
2023-01-11T21:23:04.3056739Z  distributed/pipeline/sync/test_transparency
2023-01-11T21:23:04.3057081Z  distributed/pipeline/sync/test_pipe
2023-01-11T21:23:04.3057400Z  distributed/pipeline/sync/test_inplace
2023-01-11T21:23:04.3057720Z  distributed/pipeline/sync/test_copy
2023-01-11T21:23:04.3058032Z  distributed/pipeline/sync/test_balance
2023-01-11T21:23:04.3058371Z  distributed/pipeline/sync/skip/test_stash_pop
2023-01-11T21:23:04.3058740Z  distributed/pipeline/sync/skip/test_inspect_skip_layout
2023-01-11T21:23:04.3059069Z  distributed/optim/test_named_optimizer
2023-01-11T21:23:04.3059450Z  distributed/elastic/timer/api_test
2023-01-11T21:23:04.3060002Z  distributed/_shard/test_sharder
2023-01-11T21:23:04.3061431Z  distributed/_tools/test_memory_tracker
2023-01-11T21:23:04.3062089Z  distributed/elastic/metrics/api_test
2023-01-11T21:23:04.3062785Z  distributed/elastic/utils/logging_test
2023-01-11T21:23:04.3063363Z  distributed/test_launcher
2023-01-11T21:23:04.3063649Z  distributed/checkpoint/test_planner
2023-01-11T21:23:04.3063971Z  distributed/fsdp/test_checkpoint_wrapper
2023-01-11T21:23:04.3064347Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2023-01-11T21:23:04.3064689Z  distributed/elastic/utils/distributed_test
2023-01-11T21:23:04.3065053Z  distributed/tensor/parallel/test_view_sharding_dim_change
2023-01-11T21:23:04.3065409Z  distributed/elastic/timer/local_timer_test
2023-01-11T21:23:04.3065745Z  distributed/_shard/sharded_tensor/ops/test_embedding_bag
2023-01-11T21:23:04.3066109Z  distributed/_shard/sharded_tensor/ops/test_softmax
2023-01-11T21:23:04.3066690Z  distributed/_tensor/test_view_ops
2023-01-11T21:23:04.3067398Z  distributed/fsdp/test_fsdp_input
2023-01-11T21:23:04.3067964Z  distributed/elastic/timer/local_timer_example
2023-01-11T21:23:04.3068299Z  distributed/_tensor/test_math_ops
2023-01-11T21:23:04.3068738Z  distributed/fsdp/test_fsdp_apply
2023-01-11T21:23:04.3069019Z  distributed/fsdp/test_fsdp_overlap
2023-01-11T21:23:04.3069319Z  distributed/_tensor/test_api
2023-01-11T21:23:04.3069652Z  distributed/tensor/parallel/test_parallelize_api
2023-01-11T21:23:04.3069979Z  distributed/fsdp/test_fsdp_hybrid_shard
2023-01-11T21:23:04.3070321Z  distributed/checkpoint/test_file_system_checkpoint
2023-01-11T21:23:04.3070646Z  distributed/test_c10d_spawn_ucc
2023-01-11T21:23:04.3070965Z  distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2023-01-11T21:23:04.3071369Z  distributed/_tensor/test_common_rules
2023-01-11T21:23:04.3071964Z  distributed/fsdp/test_fsdp_clip_grad_norm
2023-01-11T21:23:04.3072349Z  distributed/_composable/test_compose
2023-01-11T21:23:04.3073078Z  distributed/checkpoint/test_file_system_checkpoint_cpu
2023-01-11T21:23:04.3073837Z  distributed/algorithms/test_join
2023-01-11T21:23:04.3074322Z  distributed/test_c10d_spawn_nccl
2023-01-11T21:23:04.3074606Z  distributed/fsdp/test_fsdp_grad_acc
2023-01-11T21:23:04.3074914Z  distributed/_tensor/test_tensor_ops
2023-01-11T21:23:04.3075223Z  distributed/fsdp/test_fsdp_comm_hooks
2023-01-11T21:23:04.3075497Z  distributed/test_c10d_pypg
2023-01-11T21:23:04.3075803Z  distributed/fsdp/test_fsdp_use_orig_params
2023-01-11T21:23:04.3076138Z  distributed/fsdp/test_fsdp_mixed_precision
2023-01-11T21:23:04.3076447Z  distributed/rpc/cuda/test_tensorpipe_agent
2023-01-11T21:23:04.3226096Z Prioritized test from test file changes.
2023-01-11T21:23:04.3226739Z reordering tests for PR:
2023-01-11T21:23:04.3227616Z prioritized: ['distributed/optim/test_named_optimizer']
2023-01-11T21:23:04.3232124Z the rest: ['distributed/algorithms/quantization/test_quantization', 'distributed/test_distributed_spawn', 'distributed/rpc/test_tensorpipe_agent', 'distributed/pipeline/sync/test_transparency', 'distributed/pipeline/sync/test_pipe', 'distributed/pipeline/sync/test_inplace', 'distributed/pipeline/sync/test_copy', 'distributed/pipeline/sync/test_balance', 'distributed/pipeline/sync/skip/test_stash_pop', 'distributed/pipeline/sync/skip/test_inspect_skip_layout', 'distributed/elastic/timer/api_test', 'distributed/_shard/test_sharder', 'distributed/_tools/test_memory_tracker', 'distributed/elastic/metrics/api_test', 'distributed/elastic/utils/logging_test', 'distributed/test_launcher', 'distributed/checkpoint/test_planner', 'distributed/fsdp/test_checkpoint_wrapper', 'distributed/_shard/sharded_tensor/test_megatron_prototype', 'distributed/elastic/utils/distributed_test', 'distributed/tensor/parallel/test_view_sharding_dim_change', 'distributed/elastic/timer/local_timer_test', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag', 'distributed/_shard/sharded_tensor/ops/test_softmax', 'distributed/_tensor/test_view_ops', 'distributed/fsdp/test_fsdp_input', 'distributed/elastic/timer/local_timer_example', 'distributed/_tensor/test_math_ops', 'distributed/fsdp/test_fsdp_apply', 'distributed/fsdp/test_fsdp_overlap', 'distributed/_tensor/test_api', 'distributed/tensor/parallel/test_parallelize_api', 'distributed/fsdp/test_fsdp_hybrid_shard', 'distributed/checkpoint/test_file_system_checkpoint', 'distributed/test_c10d_spawn_ucc', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks', 'distributed/_tensor/test_common_rules', 'distributed/fsdp/test_fsdp_clip_grad_norm', 'distributed/_composable/test_compose', 'distributed/checkpoint/test_file_system_checkpoint_cpu', 'distributed/algorithms/test_join', 'distributed/test_c10d_spawn_nccl', 'distributed/fsdp/test_fsdp_grad_acc', 'distributed/_tensor/test_tensor_ops', 'distributed/fsdp/test_fsdp_comm_hooks', 'distributed/test_c10d_pypg', 'distributed/fsdp/test_fsdp_use_orig_params', 'distributed/fsdp/test_fsdp_mixed_precision', 'distributed/rpc/cuda/test_tensorpipe_agent']
2023-01-11T21:23:04.3235082Z 
2023-01-11T21:23:04.3235771Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/slow-tests.json to /var/lib/jenkins/workspace/test/.pytorch-slow-tests.json
2023-01-11T21:23:04.3450974Z Downloading https://raw.githubusercontent.com/pytorch/test-infra/generated-stats/stats/disabled-tests-condensed.json to /var/lib/jenkins/workspace/test/.pytorch-disabled-tests.json
2023-01-11T21:23:04.3625263Z parallel (file granularity) tests:
2023-01-11T21:23:04.3625774Z  
2023-01-11T21:23:04.3626220Z serial (file granularity) tests:
2023-01-11T21:23:04.3626546Z  distributed/optim/test_named_optimizer
2023-01-11T21:23:04.3626903Z  distributed/algorithms/quantization/test_quantization
2023-01-11T21:23:04.3627257Z  distributed/test_distributed_spawn
2023-01-11T21:23:04.3627647Z  distributed/rpc/test_tensorpipe_agent
2023-01-11T21:23:04.3628223Z  distributed/pipeline/sync/test_transparency
2023-01-11T21:23:04.3628856Z  distributed/pipeline/sync/test_pipe
2023-01-11T21:23:04.3629524Z  distributed/pipeline/sync/test_inplace
2023-01-11T21:23:04.3630173Z  distributed/pipeline/sync/test_copy
2023-01-11T21:23:04.3630808Z  distributed/pipeline/sync/test_balance
2023-01-11T21:23:04.3631165Z  distributed/pipeline/sync/skip/test_stash_pop
2023-01-11T21:23:04.3631518Z  distributed/pipeline/sync/skip/test_inspect_skip_layout
2023-01-11T21:23:04.3631884Z  distributed/elastic/timer/api_test
2023-01-11T21:23:04.3632180Z  distributed/_shard/test_sharder
2023-01-11T21:23:04.3632477Z  distributed/_tools/test_memory_tracker
2023-01-11T21:23:04.3632768Z  distributed/elastic/metrics/api_test
2023-01-11T21:23:04.3633077Z  distributed/elastic/utils/logging_test
2023-01-11T21:23:04.3633368Z  distributed/test_launcher
2023-01-11T21:23:04.3633642Z  distributed/checkpoint/test_planner
2023-01-11T21:23:04.3633961Z  distributed/fsdp/test_checkpoint_wrapper
2023-01-11T21:23:04.3634319Z  distributed/_shard/sharded_tensor/test_megatron_prototype
2023-01-11T21:23:04.3634657Z  distributed/elastic/utils/distributed_test
2023-01-11T21:23:04.3635018Z  distributed/tensor/parallel/test_view_sharding_dim_change
2023-01-11T21:23:04.3635369Z  distributed/elastic/timer/local_timer_test
2023-01-11T21:23:04.3635701Z  distributed/_shard/sharded_tensor/ops/test_embedding_bag
2023-01-11T21:23:04.3636067Z  distributed/_shard/sharded_tensor/ops/test_softmax
2023-01-11T21:23:04.3636390Z  distributed/_tensor/test_view_ops
2023-01-11T21:23:04.3636685Z  distributed/fsdp/test_fsdp_input
2023-01-11T21:23:04.3636984Z  distributed/elastic/timer/local_timer_example
2023-01-11T21:23:04.3637295Z  distributed/_tensor/test_math_ops
2023-01-11T21:23:04.3637590Z  distributed/fsdp/test_fsdp_apply
2023-01-11T21:23:04.3637871Z  distributed/fsdp/test_fsdp_overlap
2023-01-11T21:23:04.3638160Z  distributed/_tensor/test_api
2023-01-11T21:23:04.3638481Z  distributed/tensor/parallel/test_parallelize_api
2023-01-11T21:23:04.3638798Z  distributed/fsdp/test_fsdp_hybrid_shard
2023-01-11T21:23:04.3639139Z  distributed/checkpoint/test_file_system_checkpoint
2023-01-11T21:23:04.3639459Z  distributed/test_c10d_spawn_ucc
2023-01-11T21:23:04.3639773Z  distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2023-01-11T21:23:04.3640104Z  distributed/_tensor/test_common_rules
2023-01-11T21:23:04.3640422Z  distributed/fsdp/test_fsdp_clip_grad_norm
2023-01-11T21:23:04.3640725Z  distributed/_composable/test_compose
2023-01-11T21:23:04.3641072Z  distributed/checkpoint/test_file_system_checkpoint_cpu
2023-01-11T21:23:04.3641404Z  distributed/algorithms/test_join
2023-01-11T21:23:04.3641678Z  distributed/test_c10d_spawn_nccl
2023-01-11T21:23:04.3641975Z  distributed/fsdp/test_fsdp_grad_acc
2023-01-11T21:23:04.3642277Z  distributed/_tensor/test_tensor_ops
2023-01-11T21:23:04.3642581Z  distributed/fsdp/test_fsdp_comm_hooks
2023-01-11T21:23:04.3642853Z  distributed/test_c10d_pypg
2023-01-11T21:23:04.3643159Z  distributed/fsdp/test_fsdp_use_orig_params
2023-01-11T21:23:04.3643487Z  distributed/fsdp/test_fsdp_mixed_precision
2023-01-11T21:23:04.3643793Z  distributed/rpc/cuda/test_tensorpipe_agent
2023-01-11T21:23:06.5193069Z Ignoring disabled issues:  []
2023-01-11T21:23:06.5533377Z Ignoring disabled issues:  []
2023-01-11T21:23:06.9582286Z Running distributed/optim/test_named_optimizer ... [2023-01-11 21:23:06.957736]
2023-01-11T21:23:06.9586490Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/optim/test_named_optimizer.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:23:06.958304]
2023-01-11T21:23:09.0830396Z 
2023-01-11T21:23:09.0831193Z Expand the folded group to see the log file of distributed/optim/test_named_optimizer
2023-01-11T21:23:09.0837270Z ##[group]PRINTING LOG FILE of distributed/optim/test_named_optimizer (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_named_optimizer_bvufcwiz)
2023-01-11T21:23:09.0837674Z 
2023-01-11T21:23:09.0837984Z ##[endgroup]
2023-01-11T21:23:09.0838743Z FINISHED PRINTING LOG FILE of distributed/optim/test_named_optimizer (/var/lib/jenkins/workspace/test/test-reports/distributed-optim-test_named_optimizer_bvufcwiz)
2023-01-11T21:23:09.0839120Z 
2023-01-11T21:23:09.0839476Z Running distributed/algorithms/quantization/test_quantization ... [2023-01-11 21:23:09.083147]
2023-01-11T21:23:09.0844031Z /usr/bin/mpiexec
2023-01-11T21:23:09.0845853Z MPI not available -- MPI backend tests will be skipped
2023-01-11T21:23:09.0846471Z Map different backends to different shards for distributed/algorithms/quantization/test_quantization: {'gloo': 1, 'nccl': 2}
2023-01-11T21:23:09.0849968Z Running distributed tests for the test backend with env init_method in shard 1 of 3
2023-01-11T21:23:09.0855550Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:23:09.085246]
2023-01-11T21:23:11.1957413Z 
2023-01-11T21:23:11.1958360Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2023-01-11T21:23:11.1959615Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_xhmd5n9x)
2023-01-11T21:23:11.1960197Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:23:11.1960399Z 
2023-01-11T21:23:11.1960706Z ##[endgroup]
2023-01-11T21:23:11.1961538Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_xhmd5n9x)
2023-01-11T21:23:11.1961981Z 
2023-01-11T21:23:11.1970039Z Running distributed tests for the test backend with file init_method in shard 1 of 3
2023-01-11T21:23:11.1975283Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:23:11.197195]
2023-01-11T21:23:13.2991003Z 
2023-01-11T21:23:13.2991773Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2023-01-11T21:23:13.2992874Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_rhwqsako)
2023-01-11T21:23:13.2993463Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:23:13.2993662Z 
2023-01-11T21:23:13.2993952Z ##[endgroup]
2023-01-11T21:23:13.2994752Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_rhwqsako)
2023-01-11T21:23:13.2995198Z 
2023-01-11T21:23:13.3001542Z Shard 1: nccl should be run in 2
2023-01-11T21:23:13.3002474Z Running distributed tests for the gloo backend with env init_method in shard 1 of 3
2023-01-11T21:23:13.3009872Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:23:13.300614]
2023-01-11T21:23:35.8495963Z 
2023-01-11T21:23:35.8496958Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2023-01-11T21:23:35.8498032Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_n688227j)
2023-01-11T21:23:35.8499304Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2023-01-11T21:23:35.8500180Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2023-01-11T21:23:35.8500578Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2023-01-11T21:23:35.8500951Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2023-01-11T21:23:35.8501292Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2023-01-11T21:23:35.8501671Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2023-01-11T21:23:35.8502057Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2023-01-11T21:23:35.8502398Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:23:35.8503089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:35.8503550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:35.8504137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:35.8504596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:35.8506399Z 
2023-01-11T21:23:35.8506705Z Running tests...
2023-01-11T21:23:35.8507228Z ----------------------------------------------------------------------
2023-01-11T21:23:35.8507864Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T21:23:35.8508438Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:23:35.8508914Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1034
2023-01-11T21:23:35.8509367Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1035
2023-01-11T21:23:35.8509990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:35.8510461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:35.8511100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:35.8511581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:35.8512191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:35.8512647Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:35.8513215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:35.8513693Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:35.8514173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:35.8514663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:35.8515133Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:23:35.8515634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:23:35.8516435Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:35.8517168Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:35.8517629Z ok (3.820s)
2023-01-11T21:23:35.8517780Z 
2023-01-11T21:23:35.8518053Z ----------------------------------------------------------------------
2023-01-11T21:23:35.8518386Z Ran 1 test in 3.820s
2023-01-11T21:23:35.8518551Z 
2023-01-11T21:23:35.8518627Z OK
2023-01-11T21:23:35.8518766Z 
2023-01-11T21:23:35.8518893Z Generating XML reports...
2023-01-11T21:23:35.8519562Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212316.xml
2023-01-11T21:23:35.8520361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:35.8521001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:35.8522098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:35.8522595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:35.8522839Z 
2023-01-11T21:23:35.8522950Z Running tests...
2023-01-11T21:23:35.8523343Z ----------------------------------------------------------------------
2023-01-11T21:23:35.8523950Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T21:23:35.8525171Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:23:35.8525635Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1143
2023-01-11T21:23:35.8526087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1144
2023-01-11T21:23:35.8526716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:35.8527175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:35.8527737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:35.8528213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:35.8528794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:35.8529240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:35.8529798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:35.8530264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:35.8530708Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:35.8531172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:35.8531660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:23:35.8532161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:23:35.8532823Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:35.8533498Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:35.8533892Z ok (3.834s)
2023-01-11T21:23:35.8534038Z 
2023-01-11T21:23:35.8534306Z ----------------------------------------------------------------------
2023-01-11T21:23:35.8534615Z Ran 1 test in 3.834s
2023-01-11T21:23:35.8534777Z 
2023-01-11T21:23:35.8534871Z OK
2023-01-11T21:23:35.8535005Z 
2023-01-11T21:23:35.8535133Z Generating XML reports...
2023-01-11T21:23:35.8535945Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212322.xml
2023-01-11T21:23:35.8536713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:35.8537247Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:35.8537831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:35.8538305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:35.8538519Z 
2023-01-11T21:23:35.8538628Z Running tests...
2023-01-11T21:23:35.8539031Z ----------------------------------------------------------------------
2023-01-11T21:23:35.8539637Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T21:23:35.8540183Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2023-01-11T21:23:35.8540466Z 
2023-01-11T21:23:35.8540725Z ----------------------------------------------------------------------
2023-01-11T21:23:35.8541051Z Ran 1 test in 0.001s
2023-01-11T21:23:35.8541213Z 
2023-01-11T21:23:35.8541322Z OK (skipped=1)
2023-01-11T21:23:35.8541458Z 
2023-01-11T21:23:35.8541581Z Generating XML reports...
2023-01-11T21:23:35.8542280Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212329.xml
2023-01-11T21:23:35.8543036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:35.8543488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:35.8544046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:35.8544522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:35.8544752Z 
2023-01-11T21:23:35.8544861Z Running tests...
2023-01-11T21:23:35.8545242Z ----------------------------------------------------------------------
2023-01-11T21:23:35.8545850Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T21:23:35.8546408Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2023-01-11T21:23:35.8546693Z 
2023-01-11T21:23:35.8546953Z ----------------------------------------------------------------------
2023-01-11T21:23:35.8547259Z Ran 1 test in 0.001s
2023-01-11T21:23:35.8547419Z 
2023-01-11T21:23:35.8547528Z OK (skipped=1)
2023-01-11T21:23:35.8547681Z 
2023-01-11T21:23:35.8547805Z Generating XML reports...
2023-01-11T21:23:35.8548465Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212331.xml
2023-01-11T21:23:35.8549211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:35.8549664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:35.8550242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:35.8550695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:35.8550923Z 
2023-01-11T21:23:35.8551031Z Running tests...
2023-01-11T21:23:35.8551429Z ----------------------------------------------------------------------
2023-01-11T21:23:35.8552033Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T21:23:35.8552602Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_bfp16 (0.001s)
2023-01-11T21:23:35.8552905Z 
2023-01-11T21:23:35.8553255Z ----------------------------------------------------------------------
2023-01-11T21:23:35.8553594Z Ran 1 test in 0.001s
2023-01-11T21:23:35.8553757Z 
2023-01-11T21:23:35.8553865Z OK (skipped=1)
2023-01-11T21:23:35.8554107Z 
2023-01-11T21:23:35.8554234Z Generating XML reports...
2023-01-11T21:23:35.8554903Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212333.xml
2023-01-11T21:23:35.8555663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:35.8556100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:35.8556679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:35.8557152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:35.8557382Z 
2023-01-11T21:23:35.8557491Z Running tests...
2023-01-11T21:23:35.8557882Z ----------------------------------------------------------------------
2023-01-11T21:23:35.8558487Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T21:23:35.8559070Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_fp16 (0.001s)
2023-01-11T21:23:35.8559372Z 
2023-01-11T21:23:35.8559634Z ----------------------------------------------------------------------
2023-01-11T21:23:35.8559940Z Ran 1 test in 0.001s
2023-01-11T21:23:35.8560100Z 
2023-01-11T21:23:35.8560206Z OK (skipped=1)
2023-01-11T21:23:35.8560359Z 
2023-01-11T21:23:35.8560482Z Generating XML reports...
2023-01-11T21:23:35.8561124Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212335.xml
2023-01-11T21:23:35.8561526Z 
2023-01-11T21:23:35.8561990Z ##[endgroup]
2023-01-11T21:23:35.8562718Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_n688227j)
2023-01-11T21:23:35.8563163Z 
2023-01-11T21:23:35.8563378Z Running distributed tests for the gloo backend with file init_method in shard 1 of 3
2023-01-11T21:23:35.8564149Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:23:35.851240]
2023-01-11T21:23:58.4392595Z 
2023-01-11T21:23:58.4393117Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2023-01-11T21:23:58.4396844Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_g1qtg35d)
2023-01-11T21:23:58.4398042Z <unittest.suite.TestSuite tests=[<__main__.DistQuantizationTests testMethod=test_all_gather_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_gather_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_fp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_bfp16>, <__main__.DistQuantizationTests testMethod=test_all_to_all_single_fp16>]>
2023-01-11T21:23:58.4398933Z test_all_gather_bfp16 (__main__.DistQuantizationTests)
2023-01-11T21:23:58.4399311Z test_all_gather_fp16 (__main__.DistQuantizationTests)
2023-01-11T21:23:58.4399659Z test_all_to_all_bfp16 (__main__.DistQuantizationTests)
2023-01-11T21:23:58.4400023Z test_all_to_all_fp16 (__main__.DistQuantizationTests)
2023-01-11T21:23:58.4400406Z test_all_to_all_single_bfp16 (__main__.DistQuantizationTests)
2023-01-11T21:23:58.4400777Z test_all_to_all_single_fp16 (__main__.DistQuantizationTests)
2023-01-11T21:23:58.4401131Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:23:58.4402045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:58.4402541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:58.4403994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:58.4405189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:58.4405432Z 
2023-01-11T21:23:58.4405544Z Running tests...
2023-01-11T21:23:58.4405962Z ----------------------------------------------------------------------
2023-01-11T21:23:58.4406583Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T21:23:58.4408121Z   test_all_gather_bfp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:23:58.4409026Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1417
2023-01-11T21:23:58.4409884Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1418
2023-01-11T21:23:58.4411039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:58.4411518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:58.4412102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:58.4412586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:58.4413172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:58.4413634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:58.4414305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:58.4415129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:58.4415592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:58.4416059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:58.4416555Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:23:58.4417059Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:23:58.4417738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:58.4418424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:58.4418821Z ok (3.817s)
2023-01-11T21:23:58.4418972Z 
2023-01-11T21:23:58.4419244Z ----------------------------------------------------------------------
2023-01-11T21:23:58.4419563Z Ran 1 test in 3.817s
2023-01-11T21:23:58.4419726Z 
2023-01-11T21:23:58.4419820Z OK
2023-01-11T21:23:58.4419953Z 
2023-01-11T21:23:58.4420078Z Generating XML reports...
2023-01-11T21:23:58.4420756Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212339.xml
2023-01-11T21:23:58.4421504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:58.4421955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:58.4422537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:58.4423010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:58.4423227Z 
2023-01-11T21:23:58.4423339Z Running tests...
2023-01-11T21:23:58.4423744Z ----------------------------------------------------------------------
2023-01-11T21:23:58.4424538Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T21:23:58.4425095Z   test_all_gather_fp16 (__main__.DistQuantizationTests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:23:58.4425654Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1526
2023-01-11T21:23:58.4426106Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1527
2023-01-11T21:23:58.4426725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:58.4427162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:58.4427740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:58.4428213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:58.4428778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:58.4429227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:58.4429803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:58.4430272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:58.4430699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:23:58.4431174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:23:58.4431667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:23:58.4432146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:23:58.4432819Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:58.4433518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:23:58.4433920Z ok (3.922s)
2023-01-11T21:23:58.4434070Z 
2023-01-11T21:23:58.4434388Z ----------------------------------------------------------------------
2023-01-11T21:23:58.4434698Z Ran 1 test in 3.922s
2023-01-11T21:23:58.4434860Z 
2023-01-11T21:23:58.4434957Z OK
2023-01-11T21:23:58.4435091Z 
2023-01-11T21:23:58.4435216Z Generating XML reports...
2023-01-11T21:23:58.4435889Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212345.xml
2023-01-11T21:23:58.4436635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:58.4437088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:58.4437671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:58.4438130Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:58.4438367Z 
2023-01-11T21:23:58.4438476Z Running tests...
2023-01-11T21:23:58.4438883Z ----------------------------------------------------------------------
2023-01-11T21:23:58.4439495Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T21:23:58.4440046Z   test_all_to_all_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2023-01-11T21:23:58.4440331Z 
2023-01-11T21:23:58.4440599Z ----------------------------------------------------------------------
2023-01-11T21:23:58.4440929Z Ran 1 test in 0.001s
2023-01-11T21:23:58.4441092Z 
2023-01-11T21:23:58.4441183Z OK (skipped=1)
2023-01-11T21:23:58.4441341Z 
2023-01-11T21:23:58.4441467Z Generating XML reports...
2023-01-11T21:23:58.4442200Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212351.xml
2023-01-11T21:23:58.4442972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:58.4443461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:58.4444049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:58.4445316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:58.4445555Z 
2023-01-11T21:23:58.4445665Z Running tests...
2023-01-11T21:23:58.4446066Z ----------------------------------------------------------------------
2023-01-11T21:23:58.4446670Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T21:23:58.4447241Z   test_all_to_all_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_fp16 (0.001s)
2023-01-11T21:23:58.4447527Z 
2023-01-11T21:23:58.4447781Z ----------------------------------------------------------------------
2023-01-11T21:23:58.4448109Z Ran 1 test in 0.001s
2023-01-11T21:23:58.4448272Z 
2023-01-11T21:23:58.4448380Z OK (skipped=1)
2023-01-11T21:23:58.4448532Z 
2023-01-11T21:23:58.4448657Z Generating XML reports...
2023-01-11T21:23:58.4449309Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212353.xml
2023-01-11T21:23:58.4450076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:58.4450532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:58.4451096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:58.4451572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:58.4451808Z 
2023-01-11T21:23:58.4451916Z Running tests...
2023-01-11T21:23:58.4452323Z ----------------------------------------------------------------------
2023-01-11T21:23:58.4452915Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T21:23:58.4453501Z   test_all_to_all_single_bfp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_bfp16 (0.001s)
2023-01-11T21:23:58.4453809Z 
2023-01-11T21:23:58.4454076Z ----------------------------------------------------------------------
2023-01-11T21:23:58.4454403Z Ran 1 test in 0.001s
2023-01-11T21:23:58.4454547Z 
2023-01-11T21:23:58.4454655Z OK (skipped=1)
2023-01-11T21:23:58.4454807Z 
2023-01-11T21:23:58.4454932Z Generating XML reports...
2023-01-11T21:23:58.4455649Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212355.xml
2023-01-11T21:23:58.4456395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:23:58.4456851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:23:58.4457431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:23:58.4457902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:23:58.4458114Z 
2023-01-11T21:23:58.4458223Z Running tests...
2023-01-11T21:23:58.4458619Z ----------------------------------------------------------------------
2023-01-11T21:23:58.4459222Z Test results will be stored in test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization
2023-01-11T21:23:58.4459786Z   test_all_to_all_single_fp16 (__main__.DistQuantizationTests) ... skip: Only nccl backend supports all_to_all_single_fp16 (0.001s)
2023-01-11T21:23:58.4460090Z 
2023-01-11T21:23:58.4460467Z ----------------------------------------------------------------------
2023-01-11T21:23:58.4460807Z Ran 1 test in 0.001s
2023-01-11T21:23:58.4460968Z 
2023-01-11T21:23:58.4461136Z OK (skipped=1)
2023-01-11T21:23:58.4461290Z 
2023-01-11T21:23:58.4461397Z Generating XML reports...
2023-01-11T21:23:58.4462066Z Generated XML report: test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212357.xml
2023-01-11T21:23:58.4462471Z 
2023-01-11T21:23:58.4463004Z ##[endgroup]
2023-01-11T21:23:58.4463733Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_g1qtg35d)
2023-01-11T21:23:58.4464173Z 
2023-01-11T21:23:58.4464370Z Running distributed tests for the ucc backend with env init_method in shard 1 of 3
2023-01-11T21:23:58.4465163Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:23:58.440801]
2023-01-11T21:24:00.5372750Z 
2023-01-11T21:24:00.5373765Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2023-01-11T21:24:00.5375512Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_nfig27zp)
2023-01-11T21:24:00.5376105Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:24:00.5376310Z 
2023-01-11T21:24:00.5376605Z ##[endgroup]
2023-01-11T21:24:00.5377438Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_nfig27zp)
2023-01-11T21:24:00.5377888Z 
2023-01-11T21:24:00.5383889Z Running distributed tests for the ucc backend with file init_method in shard 1 of 3
2023-01-11T21:24:00.5388716Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/quantization/test_quantization.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:24:00.538463]
2023-01-11T21:24:02.6013268Z 
2023-01-11T21:24:02.6014080Z Expand the folded group to see the log file of distributed/algorithms/quantization/test_quantization
2023-01-11T21:24:02.6015827Z ##[group]PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_cwsilkn_)
2023-01-11T21:24:02.6016420Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:24:02.6016621Z 
2023-01-11T21:24:02.6016910Z ##[endgroup]
2023-01-11T21:24:02.6017730Z FINISHED PRINTING LOG FILE of distributed/algorithms/quantization/test_quantization (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-quantization-test_quantization_cwsilkn_)
2023-01-11T21:24:02.6018173Z 
2023-01-11T21:24:02.6027014Z Running distributed/test_distributed_spawn ... [2023-01-11 21:24:02.602332]
2023-01-11T21:24:02.6035311Z /usr/bin/mpiexec
2023-01-11T21:24:02.6036244Z MPI not available -- MPI backend tests will be skipped
2023-01-11T21:24:02.6037313Z Map different backends to different shards for distributed/test_distributed_spawn: {'gloo': 1, 'nccl': 2, 'ucc': 3}
2023-01-11T21:24:02.6039600Z Running distributed tests for the test backend with env init_method in shard 1 of 3
2023-01-11T21:24:02.6043076Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:24:02.603961]
2023-01-11T21:24:04.9949129Z 
2023-01-11T21:24:04.9950073Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2023-01-11T21:24:04.9951453Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_ybfdfvkm)
2023-01-11T21:24:04.9952202Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:24:04.9952423Z 
2023-01-11T21:24:04.9952714Z ##[endgroup]
2023-01-11T21:24:04.9953423Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_ybfdfvkm)
2023-01-11T21:24:04.9953889Z 
2023-01-11T21:24:04.9958555Z Running distributed tests for the test backend with file init_method in shard 1 of 3
2023-01-11T21:24:04.9963297Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:24:04.995965]
2023-01-11T21:24:07.3284057Z 
2023-01-11T21:24:07.3285654Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2023-01-11T21:24:07.3287028Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_alw514a4)
2023-01-11T21:24:07.3287539Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:24:07.3287744Z 
2023-01-11T21:24:07.3288262Z ##[endgroup]
2023-01-11T21:24:07.3288994Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_alw514a4)
2023-01-11T21:24:07.3289364Z 
2023-01-11T21:24:07.3294660Z Shard 1: nccl should be run in 2
2023-01-11T21:24:07.3295055Z Running distributed tests for the gloo backend with env init_method in shard 1 of 3
2023-01-11T21:24:07.3300128Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:24:07.329716]
2023-01-11T21:52:40.3880831Z 
2023-01-11T21:52:40.3881659Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2023-01-11T21:52:40.3882684Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_p9kksj0a)
2023-01-11T21:52:40.3891549Z <unittest.suite.TestSuite tests=[]>
2023-01-11T21:52:40.3929315Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_grad_as_bucket_view_false>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_ignored_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2023-01-11T21:52:40.3964988Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3966651Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3967096Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3967525Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3967968Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3968447Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3968946Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3969431Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3969952Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3970515Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3971079Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3971595Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3972130Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3972658Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3973144Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3973624Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3974092Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3974533Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3974945Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3975401Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3975885Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3976374Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3976766Z test_all_gather (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3977166Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3977694Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3978123Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3978552Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3979058Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3979480Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3979847Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3980249Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3980658Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3981042Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3981455Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3981883Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3982294Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3982715Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3983148Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3983583Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3983969Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3984395Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3984852Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3985280Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3985733Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3986179Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3986622Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3987049Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3987490Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3987912Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3988345Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3988789Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3989206Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3989627Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3990037Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3990470Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3990884Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3991289Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3991745Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3992158Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3992539Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3992947Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3993352Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3993759Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3994124Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3994515Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3994935Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3995330Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3995780Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3996183Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3996619Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3996999Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3997404Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3997809Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3998204Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3998599Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3998980Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3999352Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.3999764Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4000176Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4000588Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4000973Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4001370Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4001785Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4002202Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4002647Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4003095Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4003558Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4003999Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4004860Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4005318Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4005751Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4006197Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4006657Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4007112Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4007556Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4008033Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4008490Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4008934Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4009371Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4009771Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4010162Z test_backend_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4010513Z test_barrier (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4010886Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4011275Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4011658Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4012053Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4012451Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4012841Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4013260Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4013764Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4014185Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4014635Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4015073Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4015502Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4015905Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4016333Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4016754Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4017197Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4017612Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4018043Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4018443Z test_broadcast (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4018803Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4019204Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4019603Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4019983Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4020390Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4020857Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4021380Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4021837Z test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4022312Z test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4022808Z test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4023231Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4023660Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4024088Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4024530Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4024970Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4025455Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4025910Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4026324Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4026779Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4027202Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4027589Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4027966Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4028417Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4028838Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4029269Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4029715Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4030142Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4030564Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4031017Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4031586Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4032169Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4032867Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4033490Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4034112Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4034730Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4035351Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4035943Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4036555Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4037113Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4037618Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4038058Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4038452Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4038864Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4039270Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4039673Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4040114Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4040561Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4041016Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4041498Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4041922Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4042302Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4042724Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4043171Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4043598Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4044021Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4045089Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4045726Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4046160Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4046595Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4047013Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4047411Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4047845Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4048257Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4048791Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4049256Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4049708Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4050182Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4050555Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4050966Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4051402Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4051832Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4052202Z test_gather (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4052573Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4052957Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4053331Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4053719Z test_gather_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4054100Z test_gather_object (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4054486Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4054878Z test_get_backend (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4055249Z test_get_future (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4055615Z test_get_rank (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4055983Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4056388Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4056791Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4057150Z test_irecv (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4057503Z test_isend (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4057895Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4058319Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4058747Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4059217Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4059676Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4060082Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4060517Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4060962Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4061383Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4061815Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4062250Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4062672Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4063070Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4063492Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4063915Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4064312Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4064793Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4065296Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4065779Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4066236Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4066757Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4067238Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4067701Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4068137Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4068575Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4069027Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4069467Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4069954Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4070479Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4070955Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4071389Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4071794Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4072212Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4072607Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4073004Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4073393Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4073767Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4074164Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4074543Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4074896Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4075278Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4075679Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4076084Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4076478Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4076867Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4077247Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4077618Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4078012Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4078387Z test_scatter (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4078744Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4079132Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4079513Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4079905Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4080287Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4080674Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4081159Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4081520Z test_send_recv (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4081905Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4082335Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4082775Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4083211Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4083616Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4084031Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4084858Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4085356Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4085767Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4086228Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4086674Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4087098Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4087513Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4087908Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4088313Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4088700Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4089105Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4089566Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4090016Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4090750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4091217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4091800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4092277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4092510Z 
2023-01-11T21:52:40.4092604Z Running tests...
2023-01-11T21:52:40.4093017Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4093554Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4094140Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4094709Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 1933
2023-01-11T21:52:40.4095166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 1934
2023-01-11T21:52:40.4095780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4096216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4096796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4097270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4097852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4098285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4098860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4099330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4099771Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4100272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4100934Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4101630Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4102141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4102675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4103219Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:52:40.4104123Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:52:40.4104780Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:52:40.4105608Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:52:40.4106275Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:52:40.4106848Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:52:40.4107660Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:52:40.4108586Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:52:40.4109253Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:52:40.4109828Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:52:40.4110633Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:52:40.4111547Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:52:40.4112217Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:52:40.4112792Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T21:52:40.4113602Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:52:40.4114514Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T21:52:40.4115003Z ok (5.610s)
2023-01-11T21:52:40.4115152Z 
2023-01-11T21:52:40.4115423Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4115735Z Ran 1 test in 5.610s
2023-01-11T21:52:40.4115900Z 
2023-01-11T21:52:40.4115995Z OK
2023-01-11T21:52:40.4116132Z 
2023-01-11T21:52:40.4116259Z Generating XML reports...
2023-01-11T21:52:40.4116858Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212411.xml
2023-01-11T21:52:40.4117590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4118048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4118630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4119090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4119324Z 
2023-01-11T21:52:40.4119435Z Running tests...
2023-01-11T21:52:40.4119841Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4120375Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4120947Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2023-01-11T21:52:40.4121304Z 
2023-01-11T21:52:40.4121574Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4121902Z Ran 1 test in 0.004s
2023-01-11T21:52:40.4122063Z 
2023-01-11T21:52:40.4122154Z OK (skipped=1)
2023-01-11T21:52:40.4122309Z 
2023-01-11T21:52:40.4122436Z Generating XML reports...
2023-01-11T21:52:40.4123044Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212419.xml
2023-01-11T21:52:40.4123773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4124546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4125154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4125632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4125866Z 
2023-01-11T21:52:40.4125958Z Running tests...
2023-01-11T21:52:40.4126369Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4126903Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4127418Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4127897Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2078
2023-01-11T21:52:40.4128352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2079
2023-01-11T21:52:40.4128967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4129405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4129991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4130467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4131054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4131484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4132065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4132577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4133038Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4133521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4134192Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4134895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4135409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4135887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4136233Z ok (4.213s)
2023-01-11T21:52:40.4136384Z 
2023-01-11T21:52:40.4136653Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4136964Z Ran 1 test in 4.213s
2023-01-11T21:52:40.4137126Z 
2023-01-11T21:52:40.4137225Z OK
2023-01-11T21:52:40.4137361Z 
2023-01-11T21:52:40.4137487Z Generating XML reports...
2023-01-11T21:52:40.4138081Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212422.xml
2023-01-11T21:52:40.4138883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4139347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4139990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4140447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4140678Z 
2023-01-11T21:52:40.4140789Z Running tests...
2023-01-11T21:52:40.4141195Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4141711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4142249Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4143315Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.633s)
2023-01-11T21:52:40.4143845Z 
2023-01-11T21:52:40.4144109Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4144438Z Ran 1 test in 1.633s
2023-01-11T21:52:40.4144601Z 
2023-01-11T21:52:40.4144690Z OK (skipped=1)
2023-01-11T21:52:40.4144845Z 
2023-01-11T21:52:40.4144970Z Generating XML reports...
2023-01-11T21:52:40.4145580Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212428.xml
2023-01-11T21:52:40.4146305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4146748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4147336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4147808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4148042Z 
2023-01-11T21:52:40.4148134Z Running tests...
2023-01-11T21:52:40.4148536Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4149067Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4149617Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4150120Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2221
2023-01-11T21:52:40.4150571Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2222
2023-01-11T21:52:40.4151183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4151624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4152204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4152684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4153269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4153700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4154276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4154743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4155201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4155739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4156413Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4157165Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4157675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4158154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4158636Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4159127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4159464Z ok (4.217s)
2023-01-11T21:52:40.4159612Z 
2023-01-11T21:52:40.4159889Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4160220Z Ran 1 test in 4.218s
2023-01-11T21:52:40.4160388Z 
2023-01-11T21:52:40.4160464Z OK
2023-01-11T21:52:40.4160599Z 
2023-01-11T21:52:40.4160725Z Generating XML reports...
2023-01-11T21:52:40.4161338Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212432.xml
2023-01-11T21:52:40.4162060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4162494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4163076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4163552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4163785Z 
2023-01-11T21:52:40.4163875Z Running tests...
2023-01-11T21:52:40.4164593Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4165146Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4165711Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4166235Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2334
2023-01-11T21:52:40.4166687Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2335
2023-01-11T21:52:40.4167299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4167757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4168320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4168792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4169380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4169811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4170388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4170858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4171314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4171791Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4172449Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4173147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4173757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4174226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4174764Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4175256Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4175594Z ok (4.248s)
2023-01-11T21:52:40.4175743Z 
2023-01-11T21:52:40.4176016Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4176347Z Ran 1 test in 4.248s
2023-01-11T21:52:40.4176512Z 
2023-01-11T21:52:40.4176587Z OK
2023-01-11T21:52:40.4176721Z 
2023-01-11T21:52:40.4176846Z Generating XML reports...
2023-01-11T21:52:40.4177457Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212439.xml
2023-01-11T21:52:40.4178184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4178622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4179204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4179683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4179916Z 
2023-01-11T21:52:40.4180026Z Running tests...
2023-01-11T21:52:40.4180414Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4180950Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4181507Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4182022Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2447
2023-01-11T21:52:40.4182478Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2448
2023-01-11T21:52:40.4183090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4183550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4184110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4184585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4185165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4185596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4186167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4186636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4187098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4187579Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4188247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4188940Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4189469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4189924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4190404Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4190945Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4191474Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4191967Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4192498Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4192979Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4193438Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4193917Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4194393Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4194849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4195331Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4195808Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4196156Z ok (6.011s)
2023-01-11T21:52:40.4196290Z 
2023-01-11T21:52:40.4196569Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4196899Z Ran 1 test in 6.011s
2023-01-11T21:52:40.4197060Z 
2023-01-11T21:52:40.4197155Z OK
2023-01-11T21:52:40.4197288Z 
2023-01-11T21:52:40.4197395Z Generating XML reports...
2023-01-11T21:52:40.4198017Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212446.xml
2023-01-11T21:52:40.4198739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4199196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4199762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4200238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4200470Z 
2023-01-11T21:52:40.4200584Z Running tests...
2023-01-11T21:52:40.4200973Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4201507Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4202080Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4202627Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2562
2023-01-11T21:52:40.4203059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2563
2023-01-11T21:52:40.4203664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4204122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4204978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4205437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4206024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4206476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4207030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4207566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4208028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4208532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4209255Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4209965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4210549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4211028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4211492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4211983Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4212468Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4212932Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4213287Z ok (5.106s)
2023-01-11T21:52:40.4213435Z 
2023-01-11T21:52:40.4213705Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4214036Z Ran 1 test in 5.107s
2023-01-11T21:52:40.4214176Z 
2023-01-11T21:52:40.4214271Z OK
2023-01-11T21:52:40.4214406Z 
2023-01-11T21:52:40.4214532Z Generating XML reports...
2023-01-11T21:52:40.4215144Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212455.xml
2023-01-11T21:52:40.4215846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4216297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4216873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4217346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4217560Z 
2023-01-11T21:52:40.4217674Z Running tests...
2023-01-11T21:52:40.4218077Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4218610Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4219172Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4219723Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2677
2023-01-11T21:52:40.4220173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2678
2023-01-11T21:52:40.4220782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4221217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4221797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4222269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4222830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4223283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4223857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4224325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4224763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4225263Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4225927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4226674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4227187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4227723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4228206Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4228678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4229159Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4229639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4230113Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4230575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4231047Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4231522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4231852Z ok (5.109s)
2023-01-11T21:52:40.4232000Z 
2023-01-11T21:52:40.4232274Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4232648Z Ran 1 test in 5.109s
2023-01-11T21:52:40.4232810Z 
2023-01-11T21:52:40.4232905Z OK
2023-01-11T21:52:40.4233019Z 
2023-01-11T21:52:40.4233146Z Generating XML reports...
2023-01-11T21:52:40.4233760Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212502.xml
2023-01-11T21:52:40.4234484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4234926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4235508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4235986Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4236217Z 
2023-01-11T21:52:40.4236328Z Running tests...
2023-01-11T21:52:40.4236715Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4237245Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4237845Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4238412Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2792
2023-01-11T21:52:40.4238841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2793
2023-01-11T21:52:40.4239451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4239904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4240467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4240939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4241525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4241977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4242534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4243007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4243517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4244008Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4245092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4245794Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4246326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4246788Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4247269Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4247756Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4248109Z ok (5.235s)
2023-01-11T21:52:40.4248239Z 
2023-01-11T21:52:40.4248520Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4248855Z Ran 1 test in 5.235s
2023-01-11T21:52:40.4249016Z 
2023-01-11T21:52:40.4249115Z OK
2023-01-11T21:52:40.4249249Z 
2023-01-11T21:52:40.4249355Z Generating XML reports...
2023-01-11T21:52:40.4249966Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212510.xml
2023-01-11T21:52:40.4250690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4251146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4251710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4252181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4252413Z 
2023-01-11T21:52:40.4252522Z Running tests...
2023-01-11T21:52:40.4252910Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4253446Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4254045Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4254607Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 2907
2023-01-11T21:52:40.4255040Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 2908
2023-01-11T21:52:40.4255642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4256097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4256655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4257134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4257717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4258171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4258729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4259200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4259657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4260155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4260799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4261569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4262107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4262621Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4263104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4263593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4264078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4264537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4264887Z ok (5.750s)
2023-01-11T21:52:40.4265034Z 
2023-01-11T21:52:40.4265308Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4265619Z Ran 1 test in 5.750s
2023-01-11T21:52:40.4265786Z 
2023-01-11T21:52:40.4265882Z OK
2023-01-11T21:52:40.4266016Z 
2023-01-11T21:52:40.4266140Z Generating XML reports...
2023-01-11T21:52:40.4266751Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212518.xml
2023-01-11T21:52:40.4267460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4267912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4268491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4268970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4269184Z 
2023-01-11T21:52:40.4269293Z Running tests...
2023-01-11T21:52:40.4269701Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4270239Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4270792Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4271336Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3022
2023-01-11T21:52:40.4271786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3023
2023-01-11T21:52:40.4272398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4272833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4273412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4273885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4274454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4274901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4275473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4275946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4276384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4276887Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4277552Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4278247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4278807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4279291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4279859Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4280332Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4280815Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4281299Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4281649Z ok (5.507s)
2023-01-11T21:52:40.4281779Z 
2023-01-11T21:52:40.4282053Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4282388Z Ran 1 test in 5.507s
2023-01-11T21:52:40.4282550Z 
2023-01-11T21:52:40.4282644Z OK
2023-01-11T21:52:40.4282778Z 
2023-01-11T21:52:40.4282883Z Generating XML reports...
2023-01-11T21:52:40.4283497Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212526.xml
2023-01-11T21:52:40.4284493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4284967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4285533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4286008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4286240Z 
2023-01-11T21:52:40.4286352Z Running tests...
2023-01-11T21:52:40.4286737Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4287268Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4287864Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4288430Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3137
2023-01-11T21:52:40.4288863Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3138
2023-01-11T21:52:40.4289471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4289929Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4290491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4290962Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4291543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4291995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4292558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4293028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4293488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4293992Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4294643Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4295341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4295871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4296406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4296901Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4297397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4297937Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4298405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4298754Z ok (5.008s)
2023-01-11T21:52:40.4298903Z 
2023-01-11T21:52:40.4299174Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4299487Z Ran 1 test in 5.008s
2023-01-11T21:52:40.4299649Z 
2023-01-11T21:52:40.4299743Z OK
2023-01-11T21:52:40.4299879Z 
2023-01-11T21:52:40.4300004Z Generating XML reports...
2023-01-11T21:52:40.4300618Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212534.xml
2023-01-11T21:52:40.4301324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4301779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4302366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4302841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4303056Z 
2023-01-11T21:52:40.4303166Z Running tests...
2023-01-11T21:52:40.4303572Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4304108Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4304648Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4305724Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.628s)
2023-01-11T21:52:40.4306253Z 
2023-01-11T21:52:40.4306518Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4306855Z Ran 1 test in 1.628s
2023-01-11T21:52:40.4307018Z 
2023-01-11T21:52:40.4307108Z OK (skipped=1)
2023-01-11T21:52:40.4307262Z 
2023-01-11T21:52:40.4307388Z Generating XML reports...
2023-01-11T21:52:40.4307993Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212541.xml
2023-01-11T21:52:40.4308708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4309144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4309727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4310204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4310438Z 
2023-01-11T21:52:40.4310530Z Running tests...
2023-01-11T21:52:40.4310934Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4311463Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4312015Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4312526Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3286
2023-01-11T21:52:40.4312974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3287
2023-01-11T21:52:40.4313585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4314072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4314663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4315187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4315772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4316203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4316781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4317249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4317710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4318198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4318865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4319563Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4320071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4320546Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4320893Z ok (4.227s)
2023-01-11T21:52:40.4321043Z 
2023-01-11T21:52:40.4321311Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4321621Z Ran 1 test in 4.227s
2023-01-11T21:52:40.4321782Z 
2023-01-11T21:52:40.4321876Z OK
2023-01-11T21:52:40.4322011Z 
2023-01-11T21:52:40.4322137Z Generating XML reports...
2023-01-11T21:52:40.4322733Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212546.xml
2023-01-11T21:52:40.4323452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4323908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4324740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4325200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4325435Z 
2023-01-11T21:52:40.4325546Z Running tests...
2023-01-11T21:52:40.4325957Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4326473Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4327043Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4328121Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.626s)
2023-01-11T21:52:40.4328652Z 
2023-01-11T21:52:40.4328918Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4329246Z Ran 1 test in 1.626s
2023-01-11T21:52:40.4329390Z 
2023-01-11T21:52:40.4329500Z OK (skipped=1)
2023-01-11T21:52:40.4329654Z 
2023-01-11T21:52:40.4329779Z Generating XML reports...
2023-01-11T21:52:40.4330384Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212552.xml
2023-01-11T21:52:40.4331085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4331629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4332227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4332806Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4333019Z 
2023-01-11T21:52:40.4333131Z Running tests...
2023-01-11T21:52:40.4333538Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4334069Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4334584Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4335094Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3429
2023-01-11T21:52:40.4335542Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3430
2023-01-11T21:52:40.4336159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4336597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4337179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4337648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4338230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4338660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4339226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4339696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4340141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4340641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4341303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4341997Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4342505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4342980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4343329Z ok (5.117s)
2023-01-11T21:52:40.4343477Z 
2023-01-11T21:52:40.4343725Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4344055Z Ran 1 test in 5.117s
2023-01-11T21:52:40.4344217Z 
2023-01-11T21:52:40.4344311Z OK
2023-01-11T21:52:40.4344444Z 
2023-01-11T21:52:40.4344573Z Generating XML reports...
2023-01-11T21:52:40.4345165Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212556.xml
2023-01-11T21:52:40.4345898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4346352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4346915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4347393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4347623Z 
2023-01-11T21:52:40.4347733Z Running tests...
2023-01-11T21:52:40.4348136Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4348650Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4349215Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2023-01-11T21:52:40.4349511Z 
2023-01-11T21:52:40.4349777Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4350169Z Ran 1 test in 0.002s
2023-01-11T21:52:40.4350312Z 
2023-01-11T21:52:40.4350422Z OK (skipped=1)
2023-01-11T21:52:40.4350576Z 
2023-01-11T21:52:40.4350700Z Generating XML reports...
2023-01-11T21:52:40.4351305Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212604.xml
2023-01-11T21:52:40.4352004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4352457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4353037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4353518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4353729Z 
2023-01-11T21:52:40.4353839Z Running tests...
2023-01-11T21:52:40.4354242Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4354780Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4355222Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4355704Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4356188Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3573
2023-01-11T21:52:40.4356636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3574
2023-01-11T21:52:40.4357229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4357684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4358268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4358729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4359310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4359760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4360337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4360787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4361240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4361736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4362382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4363078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4363613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4364090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4364650Z ok (4.149s)
2023-01-11T21:52:40.4364800Z 
2023-01-11T21:52:40.4365075Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4365402Z Ran 1 test in 4.150s
2023-01-11T21:52:40.4365564Z 
2023-01-11T21:52:40.4365658Z OK
2023-01-11T21:52:40.4365773Z 
2023-01-11T21:52:40.4365898Z Generating XML reports...
2023-01-11T21:52:40.4366596Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212606.xml
2023-01-11T21:52:40.4367334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4367835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4368415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4368894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4369129Z 
2023-01-11T21:52:40.4369241Z Running tests...
2023-01-11T21:52:40.4369628Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4370161Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4370641Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4371127Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4371611Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3686
2023-01-11T21:52:40.4372065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3687
2023-01-11T21:52:40.4372675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4373110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4373689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4374162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4374725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4375177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4375756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4376226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4376669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4377167Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4377827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4378519Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4379027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4379502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4379984Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4380454Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4380812Z ok (4.227s)
2023-01-11T21:52:40.4380960Z 
2023-01-11T21:52:40.4381227Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4381555Z Ran 1 test in 4.227s
2023-01-11T21:52:40.4381697Z 
2023-01-11T21:52:40.4381792Z OK
2023-01-11T21:52:40.4381926Z 
2023-01-11T21:52:40.4382051Z Generating XML reports...
2023-01-11T21:52:40.4382658Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212613.xml
2023-01-11T21:52:40.4383361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4383815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4384446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4384928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4385185Z 
2023-01-11T21:52:40.4385295Z Running tests...
2023-01-11T21:52:40.4385705Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4386237Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4386719Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4387261Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4387765Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3799
2023-01-11T21:52:40.4388210Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3800
2023-01-11T21:52:40.4388803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4389252Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4389835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4390311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4390875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4391322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4391898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4392350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4392811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4393315Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4393984Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4394656Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4395183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4395662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4396147Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4396619Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.4396973Z ok (4.230s)
2023-01-11T21:52:40.4397124Z 
2023-01-11T21:52:40.4397395Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4397703Z Ran 1 test in 4.231s
2023-01-11T21:52:40.4397868Z 
2023-01-11T21:52:40.4397962Z OK
2023-01-11T21:52:40.4398098Z 
2023-01-11T21:52:40.4398223Z Generating XML reports...
2023-01-11T21:52:40.4398818Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212620.xml
2023-01-11T21:52:40.4399546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4399999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4400579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4401035Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4401266Z 
2023-01-11T21:52:40.4401427Z Running tests...
2023-01-11T21:52:40.4401839Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4402348Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4402878Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T21:52:40.4403374Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4403856Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 3912
2023-01-11T21:52:40.4404513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 3913
2023-01-11T21:52:40.4405140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4405597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4406180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4406639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4407225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4407677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4408236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4408709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4409168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4409667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4410321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4411020Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4411553Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4412031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4412361Z ok (4.233s)
2023-01-11T21:52:40.4412510Z 
2023-01-11T21:52:40.4412781Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4413108Z Ran 1 test in 4.233s
2023-01-11T21:52:40.4413269Z 
2023-01-11T21:52:40.4413346Z OK
2023-01-11T21:52:40.4413480Z 
2023-01-11T21:52:40.4413604Z Generating XML reports...
2023-01-11T21:52:40.4414217Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212627.xml
2023-01-11T21:52:40.4414948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4415384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4415965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4416437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4416670Z 
2023-01-11T21:52:40.4416760Z Running tests...
2023-01-11T21:52:40.4417165Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4417694Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4418195Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4418657Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4025
2023-01-11T21:52:40.4419182Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4026
2023-01-11T21:52:40.4419804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4420299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4420878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4421352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4421934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4422362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4422937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4423410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4423853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4424356Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4425022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4425706Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4426218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4426693Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4427471Z STAGE:2023-01-11 21:26:37 4026:4026 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:26:37 4025:4025 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4427851Z 
2023-01-11T21:52:40.4428397Z STAGE:2023-01-11 21:26:37 4025:4025 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:26:37 4026:4026 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4428790Z 
2023-01-11T21:52:40.4429357Z STAGE:2023-01-11 21:26:37 4025:4025 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:26:37 4026:4026 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4429759Z 
2023-01-11T21:52:40.4430067Z STAGE:2023-01-11 21:26:37 4026:4026 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4430639Z STAGE:2023-01-11 21:26:37 4025:4025 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4431214Z STAGE:2023-01-11 21:26:37 4026:4026 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4431799Z STAGE:2023-01-11 21:26:37 4025:4025 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4432377Z STAGE:2023-01-11 21:26:37 4026:4026 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4433030Z STAGE:2023-01-11 21:26:37 4025:4025 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4433391Z ok (4.210s)
2023-01-11T21:52:40.4433542Z 
2023-01-11T21:52:40.4433789Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4434118Z Ran 1 test in 4.210s
2023-01-11T21:52:40.4434274Z 
2023-01-11T21:52:40.4434369Z OK
2023-01-11T21:52:40.4434508Z 
2023-01-11T21:52:40.4434631Z Generating XML reports...
2023-01-11T21:52:40.4435240Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212633.xml
2023-01-11T21:52:40.4435945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4436450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4437037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4437568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4437781Z 
2023-01-11T21:52:40.4437888Z Running tests...
2023-01-11T21:52:40.4438291Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4438818Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4439329Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4439830Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4138
2023-01-11T21:52:40.4440269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4139
2023-01-11T21:52:40.4440880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4441311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4441893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4442363Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4442926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4443373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4443946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4444659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4445098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4445597Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4446260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4446959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4447467Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4447941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4448516Z STAGE:2023-01-11 21:26:44 4139:4139 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4449068Z STAGE:2023-01-11 21:26:44 4138:4138 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4450064Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4450696Z   warnings.warn(
2023-01-11T21:52:40.4451561Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4452176Z   warnings.warn(
2023-01-11T21:52:40.4452629Z STAGE:2023-01-11 21:26:44 4138:4138 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4453210Z STAGE:2023-01-11 21:26:44 4139:4139 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4453799Z STAGE:2023-01-11 21:26:44 4138:4138 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4454461Z STAGE:2023-01-11 21:26:44 4139:4139 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4455058Z STAGE:2023-01-11 21:26:44 4138:4138 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4455687Z STAGE:2023-01-11 21:26:44 4139:4139 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4456256Z STAGE:2023-01-11 21:26:44 4139:4139 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4456815Z STAGE:2023-01-11 21:26:44 4138:4138 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4457398Z STAGE:2023-01-11 21:26:44 4139:4139 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4457993Z STAGE:2023-01-11 21:26:44 4138:4138 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4458569Z STAGE:2023-01-11 21:26:44 4138:4138 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4459112Z STAGE:2023-01-11 21:26:44 4139:4139 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4459888Z STAGE:2023-01-11 21:26:44 4138:4138 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:26:44 4139:4139 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4460279Z 
2023-01-11T21:52:40.4460838Z STAGE:2023-01-11 21:26:44 4138:4138 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:26:44 4139:4139 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4461236Z 
2023-01-11T21:52:40.4461337Z ok (4.206s)
2023-01-11T21:52:40.4461482Z 
2023-01-11T21:52:40.4461733Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4462056Z Ran 1 test in 4.206s
2023-01-11T21:52:40.4462215Z 
2023-01-11T21:52:40.4462307Z OK
2023-01-11T21:52:40.4462437Z 
2023-01-11T21:52:40.4462543Z Generating XML reports...
2023-01-11T21:52:40.4463153Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212640.xml
2023-01-11T21:52:40.4463876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4464329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4464889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4465355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4465585Z 
2023-01-11T21:52:40.4465693Z Running tests...
2023-01-11T21:52:40.4465957Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4466273Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4466542Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4466761Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4251
2023-01-11T21:52:40.4466977Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4252
2023-01-11T21:52:40.4467350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4467527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4467907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4468099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4468470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4468643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4469060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4469259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4469560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4469797Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4470198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4470597Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4470829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4471060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4471304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.4471525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.4471928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4472259Z STAGE:2023-01-11 21:26:51 4252:4252 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4473003Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4473117Z   warnings.warn(
2023-01-11T21:52:40.4473517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4473845Z STAGE:2023-01-11 21:26:51 4251:4251 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4474587Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4474696Z   warnings.warn(
2023-01-11T21:52:40.4475010Z STAGE:2023-01-11 21:26:51 4252:4252 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4475333Z STAGE:2023-01-11 21:26:51 4251:4251 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4475677Z STAGE:2023-01-11 21:26:51 4252:4252 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4476024Z STAGE:2023-01-11 21:26:51 4251:4251 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4476347Z STAGE:2023-01-11 21:26:51 4251:4251 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4476667Z STAGE:2023-01-11 21:26:51 4252:4252 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4476993Z STAGE:2023-01-11 21:26:51 4251:4251 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4477322Z STAGE:2023-01-11 21:26:51 4252:4252 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4477658Z STAGE:2023-01-11 21:26:51 4251:4251 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4477979Z STAGE:2023-01-11 21:26:51 4252:4252 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4478300Z STAGE:2023-01-11 21:26:51 4251:4251 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4478667Z STAGE:2023-01-11 21:26:51 4252:4252 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4479011Z STAGE:2023-01-11 21:26:51 4252:4252 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4479386Z STAGE:2023-01-11 21:26:51 4251:4251 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4479723Z STAGE:2023-01-11 21:26:51 4252:4252 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4480059Z STAGE:2023-01-11 21:26:51 4251:4251 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4480159Z ok (4.245s)
2023-01-11T21:52:40.4480180Z 
2023-01-11T21:52:40.4480442Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4480535Z Ran 1 test in 4.245s
2023-01-11T21:52:40.4480554Z 
2023-01-11T21:52:40.4480646Z OK
2023-01-11T21:52:40.4480665Z 
2023-01-11T21:52:40.4480790Z Generating XML reports...
2023-01-11T21:52:40.4481249Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212647.xml
2023-01-11T21:52:40.4481625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4481804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4482187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4482379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4482399Z 
2023-01-11T21:52:40.4482506Z Running tests...
2023-01-11T21:52:40.4482754Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4483068Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4483344Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4483564Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4370
2023-01-11T21:52:40.4483770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4371
2023-01-11T21:52:40.4484144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4484548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4484938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4485115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4485483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4485663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4486043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4486232Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4486480Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4486720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4487122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4487519Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4487735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4487963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4488119Z skip: Skipped due to small world size. (4.144s)
2023-01-11T21:52:40.4488216Z 
2023-01-11T21:52:40.4488494Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4488606Z Ran 1 test in 4.144s
2023-01-11T21:52:40.4488678Z 
2023-01-11T21:52:40.4488788Z OK (skipped=1)
2023-01-11T21:52:40.4488806Z 
2023-01-11T21:52:40.4488931Z Generating XML reports...
2023-01-11T21:52:40.4489390Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212654.xml
2023-01-11T21:52:40.4489748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4489923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4490302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4490494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4490514Z 
2023-01-11T21:52:40.4490623Z Running tests...
2023-01-11T21:52:40.4490887Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4491256Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4491537Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4491755Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4479
2023-01-11T21:52:40.4491949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4480
2023-01-11T21:52:40.4492321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4492495Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4492876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4493072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4493441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4493619Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4493997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4494172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4494419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4494657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4495057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4495456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4495687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4495918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4496251Z STAGE:2023-01-11 21:27:04 4479:4479 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4496569Z STAGE:2023-01-11 21:27:04 4480:4480 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4497310Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4497408Z   warnings.warn(
2023-01-11T21:52:40.4498200Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4498355Z   warnings.warn(
2023-01-11T21:52:40.4498904Z STAGE:2023-01-11 21:27:04 4480:4480 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:27:04 4479:4479 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4498925Z 
2023-01-11T21:52:40.4499491Z STAGE:2023-01-11 21:27:04 4480:4480 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:27:04 4479:4479 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4499513Z 
2023-01-11T21:52:40.4499838Z STAGE:2023-01-11 21:27:04 4480:4480 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4500158Z STAGE:2023-01-11 21:27:04 4479:4479 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4500691Z STAGE:2023-01-11 21:27:04 4480:4480 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:27:04 4479:4479 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4500714Z 
2023-01-11T21:52:40.4501057Z STAGE:2023-01-11 21:27:04 4480:4480 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4501400Z STAGE:2023-01-11 21:27:04 4479:4479 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4501723Z STAGE:2023-01-11 21:27:04 4480:4480 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4502022Z STAGE:2023-01-11 21:27:04 4479:4479 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4502352Z STAGE:2023-01-11 21:27:04 4480:4480 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4502681Z STAGE:2023-01-11 21:27:04 4479:4479 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4503021Z STAGE:2023-01-11 21:27:04 4480:4480 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4503363Z STAGE:2023-01-11 21:27:04 4479:4479 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4503464Z ok (4.221s)
2023-01-11T21:52:40.4503483Z 
2023-01-11T21:52:40.4503745Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4503856Z Ran 1 test in 4.221s
2023-01-11T21:52:40.4503875Z 
2023-01-11T21:52:40.4503966Z OK
2023-01-11T21:52:40.4503985Z 
2023-01-11T21:52:40.4504089Z Generating XML reports...
2023-01-11T21:52:40.4504545Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212700.xml
2023-01-11T21:52:40.4504920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4505100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4505484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4505680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4505699Z 
2023-01-11T21:52:40.4505808Z Running tests...
2023-01-11T21:52:40.4506069Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4506364Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4506644Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4506863Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4592
2023-01-11T21:52:40.4507071Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4593
2023-01-11T21:52:40.4507491Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4507673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4508100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4508293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4508659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4508815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4509191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4509383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4509631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4509867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4510272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4510671Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4510903Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4511132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4511449Z STAGE:2023-01-11 21:27:11 4592:4592 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4511767Z STAGE:2023-01-11 21:27:11 4593:4593 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4512513Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4512629Z   warnings.warn(
2023-01-11T21:52:40.4513356Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4513464Z   warnings.warn(
2023-01-11T21:52:40.4513797Z STAGE:2023-01-11 21:27:11 4593:4593 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4514124Z STAGE:2023-01-11 21:27:11 4592:4592 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4514470Z STAGE:2023-01-11 21:27:11 4593:4593 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4514797Z STAGE:2023-01-11 21:27:11 4592:4592 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4514903Z ok (4.273s)
2023-01-11T21:52:40.4514924Z 
2023-01-11T21:52:40.4515190Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4515302Z Ran 1 test in 4.273s
2023-01-11T21:52:40.4515323Z 
2023-01-11T21:52:40.4515415Z OK
2023-01-11T21:52:40.4515434Z 
2023-01-11T21:52:40.4515556Z Generating XML reports...
2023-01-11T21:52:40.4516011Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212707.xml
2023-01-11T21:52:40.4516381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4516557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4517008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4517208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4517267Z 
2023-01-11T21:52:40.4517377Z Running tests...
2023-01-11T21:52:40.4517641Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4517956Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4518218Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4518436Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4705
2023-01-11T21:52:40.4518645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4706
2023-01-11T21:52:40.4519003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4519181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4519564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4519758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4520126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4520299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4520677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4520865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4521111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4521331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4521734Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4522134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4522370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4522601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4522931Z STAGE:2023-01-11 21:27:18 4706:4706 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4523252Z STAGE:2023-01-11 21:27:18 4705:4705 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4523789Z STAGE:2023-01-11 21:27:18 4705:4705 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:27:18 4706:4706 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4523813Z 
2023-01-11T21:52:40.4524621Z STAGE:2023-01-11 21:27:18 4705:4705 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:27:18 4706:4706 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4524649Z 
2023-01-11T21:52:40.4524980Z STAGE:2023-01-11 21:27:18 4705:4705 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4525278Z STAGE:2023-01-11 21:27:18 4706:4706 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4525607Z STAGE:2023-01-11 21:27:18 4705:4705 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4525938Z STAGE:2023-01-11 21:27:18 4706:4706 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4526278Z STAGE:2023-01-11 21:27:18 4705:4705 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4526690Z STAGE:2023-01-11 21:27:18 4706:4706 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4526803Z ok (4.199s)
2023-01-11T21:52:40.4526822Z 
2023-01-11T21:52:40.4527090Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4527258Z Ran 1 test in 4.199s
2023-01-11T21:52:40.4527277Z 
2023-01-11T21:52:40.4527370Z OK
2023-01-11T21:52:40.4527389Z 
2023-01-11T21:52:40.4527496Z Generating XML reports...
2023-01-11T21:52:40.4527953Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212714.xml
2023-01-11T21:52:40.4528327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4528506Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4528888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4529083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4529103Z 
2023-01-11T21:52:40.4529210Z Running tests...
2023-01-11T21:52:40.4529474Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4529773Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4530035Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2023-01-11T21:52:40.4530054Z 
2023-01-11T21:52:40.4530314Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4530422Z Ran 1 test in 0.002s
2023-01-11T21:52:40.4530441Z 
2023-01-11T21:52:40.4530547Z OK (skipped=1)
2023-01-11T21:52:40.4530566Z 
2023-01-11T21:52:40.4530687Z Generating XML reports...
2023-01-11T21:52:40.4531139Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212721.xml
2023-01-11T21:52:40.4531513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4531687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4532056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4532247Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4532267Z 
2023-01-11T21:52:40.4532373Z Running tests...
2023-01-11T21:52:40.4532680Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4532997Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4533270Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2023-01-11T21:52:40.4533290Z 
2023-01-11T21:52:40.4533547Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4533661Z Ran 1 test in 0.002s
2023-01-11T21:52:40.4533680Z 
2023-01-11T21:52:40.4533786Z OK (skipped=1)
2023-01-11T21:52:40.4533805Z 
2023-01-11T21:52:40.4533910Z Generating XML reports...
2023-01-11T21:52:40.4534367Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212723.xml
2023-01-11T21:52:40.4534736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4534912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4535290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4535479Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4535499Z 
2023-01-11T21:52:40.4535604Z Running tests...
2023-01-11T21:52:40.4535861Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4536224Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4536479Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4536754Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 4884
2023-01-11T21:52:40.4536965Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 4885
2023-01-11T21:52:40.4537337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4537512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4537888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4538078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4538449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4538608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4538984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4539171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4539417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4539655Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4540058Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4540453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4540687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4540916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4541146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.4541379Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.4541778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4542172Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4542503Z STAGE:2023-01-11 21:27:29 4884:4884 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4542825Z STAGE:2023-01-11 21:27:29 4885:4885 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4543366Z STAGE:2023-01-11 21:27:29 4885:4885 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:27:29 4884:4884 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4543389Z 
2023-01-11T21:52:40.4543956Z STAGE:2023-01-11 21:27:29 4885:4885 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:27:29 4884:4884 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4543976Z 
2023-01-11T21:52:40.4544296Z STAGE:2023-01-11 21:27:29 4885:4885 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4544611Z STAGE:2023-01-11 21:27:29 4884:4884 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4544939Z STAGE:2023-01-11 21:27:29 4884:4884 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4545255Z STAGE:2023-01-11 21:27:29 4885:4885 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4545654Z STAGE:2023-01-11 21:27:29 4884:4884 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4546009Z STAGE:2023-01-11 21:27:29 4885:4885 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4546154Z ok (4.318s)
2023-01-11T21:52:40.4546174Z 
2023-01-11T21:52:40.4546440Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4546550Z Ran 1 test in 4.318s
2023-01-11T21:52:40.4546569Z 
2023-01-11T21:52:40.4546661Z OK
2023-01-11T21:52:40.4546680Z 
2023-01-11T21:52:40.4546802Z Generating XML reports...
2023-01-11T21:52:40.4547238Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212725.xml
2023-01-11T21:52:40.4547610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4547784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4548164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4588193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4588233Z 
2023-01-11T21:52:40.4588368Z Running tests...
2023-01-11T21:52:40.4588688Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4589015Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4589280Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4589505Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5003
2023-01-11T21:52:40.4589724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5004
2023-01-11T21:52:40.4590096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4590277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4590667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4590864Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4591232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4591409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4591786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4591976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4592227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4592459Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4592865Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4593272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4593505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4593736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4593896Z skip: Skipped due to small world size. (4.216s)
2023-01-11T21:52:40.4593916Z 
2023-01-11T21:52:40.4594184Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4594298Z Ran 1 test in 4.216s
2023-01-11T21:52:40.4594318Z 
2023-01-11T21:52:40.4594426Z OK (skipped=1)
2023-01-11T21:52:40.4594445Z 
2023-01-11T21:52:40.4594691Z Generating XML reports...
2023-01-11T21:52:40.4595170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212732.xml
2023-01-11T21:52:40.4595616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4595794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4596176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4596368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4596389Z 
2023-01-11T21:52:40.4596498Z Running tests...
2023-01-11T21:52:40.4596761Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4597061Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4597365Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2023-01-11T21:52:40.4597386Z 
2023-01-11T21:52:40.4597647Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4597764Z Ran 1 test in 0.002s
2023-01-11T21:52:40.4597784Z 
2023-01-11T21:52:40.4597892Z OK (skipped=1)
2023-01-11T21:52:40.4597911Z 
2023-01-11T21:52:40.4598033Z Generating XML reports...
2023-01-11T21:52:40.4598491Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212739.xml
2023-01-11T21:52:40.4598863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4599040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4599403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4599600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4599619Z 
2023-01-11T21:52:40.4599728Z Running tests...
2023-01-11T21:52:40.4599995Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4600319Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4600620Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2023-01-11T21:52:40.4600640Z 
2023-01-11T21:52:40.4600899Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4601012Z Ran 1 test in 0.002s
2023-01-11T21:52:40.4601032Z 
2023-01-11T21:52:40.4601140Z OK (skipped=1)
2023-01-11T21:52:40.4601160Z 
2023-01-11T21:52:40.4601266Z Generating XML reports...
2023-01-11T21:52:40.4601718Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212741.xml
2023-01-11T21:52:40.4602098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4602275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4602663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4602855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4602875Z 
2023-01-11T21:52:40.4602983Z Running tests...
2023-01-11T21:52:40.4603244Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4603560Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4603831Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2023-01-11T21:52:40.4603850Z 
2023-01-11T21:52:40.4604162Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4604496Z Ran 1 test in 0.002s
2023-01-11T21:52:40.4604518Z 
2023-01-11T21:52:40.4604628Z OK (skipped=1)
2023-01-11T21:52:40.4604716Z 
2023-01-11T21:52:40.4604847Z Generating XML reports...
2023-01-11T21:52:40.4605317Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212744.xml
2023-01-11T21:52:40.4605689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4605866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4606246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4606422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4606441Z 
2023-01-11T21:52:40.4606548Z Running tests...
2023-01-11T21:52:40.4606815Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4607130Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4607433Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2023-01-11T21:52:40.4607453Z 
2023-01-11T21:52:40.4607713Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4607825Z Ran 1 test in 0.002s
2023-01-11T21:52:40.4607843Z 
2023-01-11T21:52:40.4607949Z OK (skipped=1)
2023-01-11T21:52:40.4607968Z 
2023-01-11T21:52:40.4608091Z Generating XML reports...
2023-01-11T21:52:40.4608527Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212746.xml
2023-01-11T21:52:40.4608902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4609083Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4609468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4609665Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4609684Z 
2023-01-11T21:52:40.4609793Z Running tests...
2023-01-11T21:52:40.4610053Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4610368Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4610629Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4610850Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5244
2023-01-11T21:52:40.4611065Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5245
2023-01-11T21:52:40.4611437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4611615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4612000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4612194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4612556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4612728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4613091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4613280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4613594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4613852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4614259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4614711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4614944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4615173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4615258Z ok (4.239s)
2023-01-11T21:52:40.4615295Z 
2023-01-11T21:52:40.4615544Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4615653Z Ran 1 test in 4.239s
2023-01-11T21:52:40.4615672Z 
2023-01-11T21:52:40.4615766Z OK
2023-01-11T21:52:40.4615785Z 
2023-01-11T21:52:40.4615912Z Generating XML reports...
2023-01-11T21:52:40.4616369Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212749.xml
2023-01-11T21:52:40.4616745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4616921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4617302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4617478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4617498Z 
2023-01-11T21:52:40.4617605Z Running tests...
2023-01-11T21:52:40.4617867Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4618182Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4618460Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4618677Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5353
2023-01-11T21:52:40.4618898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5354
2023-01-11T21:52:40.4619274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4619432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4619816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4620007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4620369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4620548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4620923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4621119Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4621367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4621613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4621999Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4622400Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4622631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4622913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4623165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.4623447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.4623846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4624238Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4624479Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:52:40.4624700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:52:40.4625096Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:52:40.4625493Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:52:40.4625735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T21:52:40.4625976Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T21:52:40.4626367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2023-01-11T21:52:40.4626757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2023-01-11T21:52:40.4626860Z ok (4.323s)
2023-01-11T21:52:40.4626880Z 
2023-01-11T21:52:40.4627147Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4627240Z Ran 1 test in 4.323s
2023-01-11T21:52:40.4627276Z 
2023-01-11T21:52:40.4627351Z OK
2023-01-11T21:52:40.4627370Z 
2023-01-11T21:52:40.4627497Z Generating XML reports...
2023-01-11T21:52:40.4627954Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212755.xml
2023-01-11T21:52:40.4628330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4628507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4628888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4629080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4629100Z 
2023-01-11T21:52:40.4629208Z Running tests...
2023-01-11T21:52:40.4629451Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4629762Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4630028Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s)
2023-01-11T21:52:40.4630048Z 
2023-01-11T21:52:40.4630304Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4630418Z Ran 1 test in 0.002s
2023-01-11T21:52:40.4630438Z 
2023-01-11T21:52:40.4630543Z OK (skipped=1)
2023-01-11T21:52:40.4630562Z 
2023-01-11T21:52:40.4630685Z Generating XML reports...
2023-01-11T21:52:40.4631137Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212802.xml
2023-01-11T21:52:40.4631511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4631671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4632052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4632300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4632322Z 
2023-01-11T21:52:40.4632435Z Running tests...
2023-01-11T21:52:40.4632756Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4633121Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4633407Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4633627Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5519
2023-01-11T21:52:40.4633828Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5520
2023-01-11T21:52:40.4634196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4634372Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4634760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4634954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4635323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4635499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4635876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4636064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4636294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4636699Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4636945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4637339Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4637574Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4637805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4638048Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.4638294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.4638691Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4639071Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4639407Z STAGE:2023-01-11 21:28:08 5519:5519 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4639737Z STAGE:2023-01-11 21:28:08 5520:5520 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4640491Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4640606Z   warnings.warn(
2023-01-11T21:52:40.4641345Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4641456Z   warnings.warn(
2023-01-11T21:52:40.4641848Z STAGE:2023-01-11 21:28:08 5519:5519 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4642189Z STAGE:2023-01-11 21:28:08 5520:5520 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4642583Z STAGE:2023-01-11 21:28:08 5519:5519 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4642930Z STAGE:2023-01-11 21:28:08 5520:5520 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4643252Z STAGE:2023-01-11 21:28:08 5520:5520 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4643575Z STAGE:2023-01-11 21:28:08 5519:5519 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4644118Z STAGE:2023-01-11 21:28:08 5520:5520 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:28:08 5519:5519 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4644139Z 
2023-01-11T21:52:40.4644685Z STAGE:2023-01-11 21:28:08 5520:5520 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4645032Z STAGE:2023-01-11 21:28:08 5519:5519 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4645141Z ok (4.239s)
2023-01-11T21:52:40.4645161Z 
2023-01-11T21:52:40.4645425Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4645537Z Ran 1 test in 4.239s
2023-01-11T21:52:40.4645557Z 
2023-01-11T21:52:40.4645632Z OK
2023-01-11T21:52:40.4645651Z 
2023-01-11T21:52:40.4645775Z Generating XML reports...
2023-01-11T21:52:40.4646233Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212805.xml
2023-01-11T21:52:40.4646605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4646782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4647168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4647364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4647387Z 
2023-01-11T21:52:40.4647495Z Running tests...
2023-01-11T21:52:40.4647740Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4648063Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4648349Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4648570Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5638
2023-01-11T21:52:40.4648787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5639
2023-01-11T21:52:40.4649166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4649344Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4649725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4649923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4650270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4650444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4650817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4651007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4651254Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4651578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4651997Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4652459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4652692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4652905Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4653145Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.4653388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.4653787Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4654187Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4654523Z STAGE:2023-01-11 21:28:15 5638:5638 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4654853Z STAGE:2023-01-11 21:28:15 5639:5639 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4655604Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4655719Z   warnings.warn(
2023-01-11T21:52:40.4656462Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4656557Z   warnings.warn(
2023-01-11T21:52:40.4656894Z STAGE:2023-01-11 21:28:15 5638:5638 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4657224Z STAGE:2023-01-11 21:28:15 5639:5639 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4657569Z STAGE:2023-01-11 21:28:15 5638:5638 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4657912Z STAGE:2023-01-11 21:28:15 5639:5639 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4658237Z STAGE:2023-01-11 21:28:15 5638:5638 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4658561Z STAGE:2023-01-11 21:28:15 5639:5639 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4658890Z STAGE:2023-01-11 21:28:15 5638:5638 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4659204Z STAGE:2023-01-11 21:28:15 5639:5639 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4659547Z STAGE:2023-01-11 21:28:15 5638:5638 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4659890Z STAGE:2023-01-11 21:28:15 5639:5639 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4659993Z ok (4.233s)
2023-01-11T21:52:40.4660013Z 
2023-01-11T21:52:40.4660276Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4660388Z Ran 1 test in 4.233s
2023-01-11T21:52:40.4660407Z 
2023-01-11T21:52:40.4660499Z OK
2023-01-11T21:52:40.4660518Z 
2023-01-11T21:52:40.4660642Z Generating XML reports...
2023-01-11T21:52:40.4661097Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212811.xml
2023-01-11T21:52:40.4661507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4661692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4662082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4662322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4662342Z 
2023-01-11T21:52:40.4662450Z Running tests...
2023-01-11T21:52:40.4662718Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4663033Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4663328Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4663530Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5757
2023-01-11T21:52:40.4663752Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5758
2023-01-11T21:52:40.4664123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4664301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4664681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4664873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4665237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4665411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4665784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4665953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4666205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4666452Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4666860Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4667260Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4667492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4667721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4667962Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.4668209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.4668589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4668984Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4669314Z STAGE:2023-01-11 21:28:22 5757:5757 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4669642Z STAGE:2023-01-11 21:28:22 5758:5758 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4670383Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4670497Z   warnings.warn(
2023-01-11T21:52:40.4671283Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4671438Z   warnings.warn(
2023-01-11T21:52:40.4671780Z STAGE:2023-01-11 21:28:22 5758:5758 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4672105Z STAGE:2023-01-11 21:28:22 5757:5757 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4672433Z STAGE:2023-01-11 21:28:22 5758:5758 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4672774Z STAGE:2023-01-11 21:28:22 5757:5757 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4673301Z STAGE:2023-01-11 21:28:22 5757:5757 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:28:22 5758:5758 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4673326Z 
2023-01-11T21:52:40.4673650Z STAGE:2023-01-11 21:28:22 5758:5758 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4673985Z STAGE:2023-01-11 21:28:22 5757:5757 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4674326Z STAGE:2023-01-11 21:28:22 5758:5758 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4674665Z STAGE:2023-01-11 21:28:22 5757:5757 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4674767Z ok (4.250s)
2023-01-11T21:52:40.4674787Z 
2023-01-11T21:52:40.4675054Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4675148Z Ran 1 test in 4.250s
2023-01-11T21:52:40.4675185Z 
2023-01-11T21:52:40.4675260Z OK
2023-01-11T21:52:40.4675279Z 
2023-01-11T21:52:40.4675401Z Generating XML reports...
2023-01-11T21:52:40.4675862Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212818.xml
2023-01-11T21:52:40.4676236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4676417Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4676799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4676993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4677013Z 
2023-01-11T21:52:40.4677120Z Running tests...
2023-01-11T21:52:40.4677366Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4677679Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4677966Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4678190Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5876
2023-01-11T21:52:40.4678409Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5877
2023-01-11T21:52:40.4678782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4678955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4679335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4679508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4679872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4680044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4680473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4680669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4680918Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4681210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4681617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4682017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4682233Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4682461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4682708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.4682953Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.4683358Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4683757Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4684090Z STAGE:2023-01-11 21:28:29 5877:5877 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4684711Z STAGE:2023-01-11 21:28:29 5876:5876 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4685472Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4685588Z   warnings.warn(
2023-01-11T21:52:40.4686309Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4686423Z   warnings.warn(
2023-01-11T21:52:40.4686971Z STAGE:2023-01-11 21:28:29 5876:5876 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:28:29 5877:5877 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4686992Z 
2023-01-11T21:52:40.4687335Z STAGE:2023-01-11 21:28:29 5877:5877 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4687678Z STAGE:2023-01-11 21:28:29 5876:5876 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4688004Z STAGE:2023-01-11 21:28:29 5876:5876 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4688325Z STAGE:2023-01-11 21:28:29 5877:5877 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4688658Z STAGE:2023-01-11 21:28:29 5877:5877 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4688987Z STAGE:2023-01-11 21:28:29 5876:5876 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4689328Z STAGE:2023-01-11 21:28:29 5877:5877 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4689653Z STAGE:2023-01-11 21:28:29 5876:5876 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4689755Z ok (4.337s)
2023-01-11T21:52:40.4689775Z 
2023-01-11T21:52:40.4690038Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4690150Z Ran 1 test in 4.337s
2023-01-11T21:52:40.4690170Z 
2023-01-11T21:52:40.4690261Z OK
2023-01-11T21:52:40.4690280Z 
2023-01-11T21:52:40.4690482Z Generating XML reports...
2023-01-11T21:52:40.4690955Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212825.xml
2023-01-11T21:52:40.4691385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4691544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4691925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4692117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4692137Z 
2023-01-11T21:52:40.4692246Z Running tests...
2023-01-11T21:52:40.4692510Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4692824Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4693113Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4693336Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 5995
2023-01-11T21:52:40.4693539Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 5996
2023-01-11T21:52:40.4693916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4694091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4694470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4694663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4695030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4695205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4695581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4695773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4696005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4696251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4696653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4697050Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4697282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4697513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4697671Z skip: Skipped due to small world size. (4.136s)
2023-01-11T21:52:40.4697692Z 
2023-01-11T21:52:40.4697965Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4698075Z Ran 1 test in 4.136s
2023-01-11T21:52:40.4698095Z 
2023-01-11T21:52:40.4698185Z OK (skipped=1)
2023-01-11T21:52:40.4698204Z 
2023-01-11T21:52:40.4698327Z Generating XML reports...
2023-01-11T21:52:40.4698784Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212832.xml
2023-01-11T21:52:40.4699153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4699326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4699707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4699948Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4699970Z 
2023-01-11T21:52:40.4700079Z Running tests...
2023-01-11T21:52:40.4700405Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4700711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4700989Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4701204Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6104
2023-01-11T21:52:40.4701418Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6105
2023-01-11T21:52:40.4701786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4701957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4702340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4702529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4702882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4703050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4703424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4703608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4703844Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4704085Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4704479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4704867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4705091Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4705304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4705452Z skip: Skipped due to small world size. (4.143s)
2023-01-11T21:52:40.4705472Z 
2023-01-11T21:52:40.4705726Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4705825Z Ran 1 test in 4.143s
2023-01-11T21:52:40.4705845Z 
2023-01-11T21:52:40.4705940Z OK (skipped=1)
2023-01-11T21:52:40.4705959Z 
2023-01-11T21:52:40.4706072Z Generating XML reports...
2023-01-11T21:52:40.4706518Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212839.xml
2023-01-11T21:52:40.4706876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4707038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4707416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4707596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4707616Z 
2023-01-11T21:52:40.4707711Z Running tests...
2023-01-11T21:52:40.4707962Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4708266Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4708544Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4708802Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6213
2023-01-11T21:52:40.4709013Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6214
2023-01-11T21:52:40.4709426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4709591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4709964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4710152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4710507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4710668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4711036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4711215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4711445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4711682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4712073Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4712466Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4712695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4712919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4713065Z skip: Skipped due to small world size. (4.211s)
2023-01-11T21:52:40.4713085Z 
2023-01-11T21:52:40.4713340Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4713441Z Ran 1 test in 4.211s
2023-01-11T21:52:40.4713460Z 
2023-01-11T21:52:40.4713553Z OK (skipped=1)
2023-01-11T21:52:40.4713580Z 
2023-01-11T21:52:40.4713687Z Generating XML reports...
2023-01-11T21:52:40.4714140Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212845.xml
2023-01-11T21:52:40.4714514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4714690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4715071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4715267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4715286Z 
2023-01-11T21:52:40.4715392Z Running tests...
2023-01-11T21:52:40.4715644Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4715944Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4716221Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4716442Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6322
2023-01-11T21:52:40.4716658Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6323
2023-01-11T21:52:40.4717033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4717207Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4717585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4717826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4718186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4718404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4718775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4718955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4719192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4719429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4719824Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4720213Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4720435Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4720651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4720801Z skip: Skipped due to small world size. (4.098s)
2023-01-11T21:52:40.4720820Z 
2023-01-11T21:52:40.4721080Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4721194Z Ran 1 test in 4.098s
2023-01-11T21:52:40.4721213Z 
2023-01-11T21:52:40.4721319Z OK (skipped=1)
2023-01-11T21:52:40.4721338Z 
2023-01-11T21:52:40.4721460Z Generating XML reports...
2023-01-11T21:52:40.4721916Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212852.xml
2023-01-11T21:52:40.4722291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4722468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4722833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4723019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4723038Z 
2023-01-11T21:52:40.4723134Z Running tests...
2023-01-11T21:52:40.4723384Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4723689Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4723951Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4724158Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6431
2023-01-11T21:52:40.4724607Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6432
2023-01-11T21:52:40.4724984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4725151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4725524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4725704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4726059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4726224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4726587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4726765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4727076Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4727476Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4727766Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4728150Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4728370Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4728589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4728911Z STAGE:2023-01-11 21:29:02 6431:6431 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4729233Z STAGE:2023-01-11 21:29:02 6432:6432 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4729972Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4730081Z   warnings.warn(
2023-01-11T21:52:40.4730809Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4730904Z   warnings.warn(
2023-01-11T21:52:40.4731439Z STAGE:2023-01-11 21:29:02 6431:6431 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:29:02 6432:6432 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4731460Z 
2023-01-11T21:52:40.4732022Z STAGE:2023-01-11 21:29:02 6431:6431 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:29:02 6432:6432 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4732045Z 
2023-01-11T21:52:40.4732358Z STAGE:2023-01-11 21:29:02 6432:6432 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4732716Z STAGE:2023-01-11 21:29:02 6431:6431 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4733250Z STAGE:2023-01-11 21:29:02 6431:6431 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:29:02 6432:6432 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4733270Z 
2023-01-11T21:52:40.4733824Z STAGE:2023-01-11 21:29:02 6431:6431 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:29:02 6432:6432 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4733843Z 
2023-01-11T21:52:40.4733933Z ok (4.209s)
2023-01-11T21:52:40.4733956Z 
2023-01-11T21:52:40.4734210Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4734309Z Ran 1 test in 4.210s
2023-01-11T21:52:40.4734332Z 
2023-01-11T21:52:40.4734413Z OK
2023-01-11T21:52:40.4734432Z 
2023-01-11T21:52:40.4734538Z Generating XML reports...
2023-01-11T21:52:40.4734985Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212859.xml
2023-01-11T21:52:40.4735346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4735512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4735883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4736065Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4736084Z 
2023-01-11T21:52:40.4736252Z Running tests...
2023-01-11T21:52:40.4736510Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4736815Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4737148Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4737358Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6544
2023-01-11T21:52:40.4737563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6545
2023-01-11T21:52:40.4737929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4738094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4738465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4738649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4739001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4739166Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4739523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4739703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4739943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4740178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4740570Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4740962Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4741186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4741409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4742140Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4742237Z   warnings.warn(
2023-01-11T21:52:40.4742965Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4743068Z   warnings.warn(
2023-01-11T21:52:40.4743158Z ok (4.136s)
2023-01-11T21:52:40.4743178Z 
2023-01-11T21:52:40.4743431Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4743536Z Ran 1 test in 4.136s
2023-01-11T21:52:40.4743555Z 
2023-01-11T21:52:40.4743637Z OK
2023-01-11T21:52:40.4743656Z 
2023-01-11T21:52:40.4743767Z Generating XML reports...
2023-01-11T21:52:40.4744210Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212905.xml
2023-01-11T21:52:40.4744561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4744729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4745098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4745328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4745349Z 
2023-01-11T21:52:40.4745450Z Running tests...
2023-01-11T21:52:40.4745707Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4746070Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4746341Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4746542Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6653
2023-01-11T21:52:40.4746759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6654
2023-01-11T21:52:40.4747128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4747305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4747678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4747853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4748223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4748408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4748775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4748949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4749184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4749427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4749831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4750227Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4750460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4750688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4751022Z STAGE:2023-01-11 21:29:16 6653:6653 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4751348Z STAGE:2023-01-11 21:29:16 6654:6654 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4752076Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4752192Z   warnings.warn(
2023-01-11T21:52:40.4752930Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4753042Z   warnings.warn(
2023-01-11T21:52:40.4753375Z STAGE:2023-01-11 21:29:16 6653:6653 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4753700Z STAGE:2023-01-11 21:29:16 6654:6654 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4754045Z STAGE:2023-01-11 21:29:16 6653:6653 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4754388Z STAGE:2023-01-11 21:29:16 6654:6654 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4754965Z STAGE:2023-01-11 21:29:16 6653:6653 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:29:16 6654:6654 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4754988Z 
2023-01-11T21:52:40.4755313Z STAGE:2023-01-11 21:29:16 6654:6654 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4755677Z STAGE:2023-01-11 21:29:16 6653:6653 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4756013Z STAGE:2023-01-11 21:29:16 6654:6654 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4756350Z STAGE:2023-01-11 21:29:16 6653:6653 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4756450Z ok (4.232s)
2023-01-11T21:52:40.4756469Z 
2023-01-11T21:52:40.4756734Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4756843Z Ran 1 test in 4.232s
2023-01-11T21:52:40.4756863Z 
2023-01-11T21:52:40.4756956Z OK
2023-01-11T21:52:40.4756975Z 
2023-01-11T21:52:40.4757098Z Generating XML reports...
2023-01-11T21:52:40.4757551Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212912.xml
2023-01-11T21:52:40.4757910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4758088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4758472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4758667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4758686Z 
2023-01-11T21:52:40.4758794Z Running tests...
2023-01-11T21:52:40.4759058Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4759371Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4759654Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4759857Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6766
2023-01-11T21:52:40.4760078Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6767
2023-01-11T21:52:40.4760451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4760626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4761000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4761188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4761552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4761725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4762103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4762275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4762518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4762760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4763164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4763561Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4763792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4764022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4764663Z STAGE:2023-01-11 21:29:23 6766:6766 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4765017Z STAGE:2023-01-11 21:29:23 6767:6767 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4765806Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4765922Z   warnings.warn(
2023-01-11T21:52:40.4766657Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4766771Z   warnings.warn(
2023-01-11T21:52:40.4767319Z STAGE:2023-01-11 21:29:23 6766:6766 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:29:23 6767:6767 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4767343Z 
2023-01-11T21:52:40.4767913Z STAGE:2023-01-11 21:29:23 6766:6766 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:29:23 6767:6767 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4767933Z 
2023-01-11T21:52:40.4768457Z STAGE:2023-01-11 21:29:23 6767:6767 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:29:23 6766:6766 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4768477Z 
2023-01-11T21:52:40.4768807Z STAGE:2023-01-11 21:29:23 6766:6766 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4769129Z STAGE:2023-01-11 21:29:23 6767:6767 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4769469Z STAGE:2023-01-11 21:29:23 6766:6766 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4769810Z STAGE:2023-01-11 21:29:23 6767:6767 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4769897Z ok (4.228s)
2023-01-11T21:52:40.4769935Z 
2023-01-11T21:52:40.4770182Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4770282Z Ran 1 test in 4.229s
2023-01-11T21:52:40.4770303Z 
2023-01-11T21:52:40.4770391Z OK
2023-01-11T21:52:40.4770410Z 
2023-01-11T21:52:40.4770534Z Generating XML reports...
2023-01-11T21:52:40.4770988Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212919.xml
2023-01-11T21:52:40.4771352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4771527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4771912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4772087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4772117Z 
2023-01-11T21:52:40.4772208Z Running tests...
2023-01-11T21:52:40.4772472Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4772789Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4773062Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4773272Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6879
2023-01-11T21:52:40.4773487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6880
2023-01-11T21:52:40.4773859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4774075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4774447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4774697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4775063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4775227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4775598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4775786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4776024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4776273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4776662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4777068Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4777290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4777516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4777848Z STAGE:2023-01-11 21:29:29 6880:6880 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4778168Z STAGE:2023-01-11 21:29:29 6879:6879 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4778911Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4779028Z   warnings.warn(
2023-01-11T21:52:40.4779762Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T21:52:40.4779874Z   warnings.warn(
2023-01-11T21:52:40.4780398Z STAGE:2023-01-11 21:29:29 6879:6879 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:29:29 6880:6880 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4780429Z 
2023-01-11T21:52:40.4780978Z STAGE:2023-01-11 21:29:29 6880:6880 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:29:29 6879:6879 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4781015Z 
2023-01-11T21:52:40.4781522Z STAGE:2023-01-11 21:29:29 6880:6880 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:29:29 6879:6879 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4781561Z 
2023-01-11T21:52:40.4781877Z STAGE:2023-01-11 21:29:29 6879:6879 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4782202Z STAGE:2023-01-11 21:29:29 6880:6880 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4782544Z STAGE:2023-01-11 21:29:29 6879:6879 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4782876Z STAGE:2023-01-11 21:29:29 6880:6880 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4782978Z ok (4.238s)
2023-01-11T21:52:40.4782997Z 
2023-01-11T21:52:40.4783264Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4783416Z Ran 1 test in 4.239s
2023-01-11T21:52:40.4783438Z 
2023-01-11T21:52:40.4783532Z OK
2023-01-11T21:52:40.4783551Z 
2023-01-11T21:52:40.4783659Z Generating XML reports...
2023-01-11T21:52:40.4784164Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212925.xml
2023-01-11T21:52:40.4784529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4784703Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4785089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4785281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4785301Z 
2023-01-11T21:52:40.4785399Z Running tests...
2023-01-11T21:52:40.4785659Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4785960Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4786249Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4786464Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 6992
2023-01-11T21:52:40.4786677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 6993
2023-01-11T21:52:40.4787048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4787215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4787596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4787789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4788153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4788310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4788687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4788874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4789115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4789359Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4789764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4790162Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4790389Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4790616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4790704Z ok (4.207s)
2023-01-11T21:52:40.4790724Z 
2023-01-11T21:52:40.4791045Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4791151Z Ran 1 test in 4.207s
2023-01-11T21:52:40.4791171Z 
2023-01-11T21:52:40.4791258Z OK
2023-01-11T21:52:40.4791277Z 
2023-01-11T21:52:40.4791401Z Generating XML reports...
2023-01-11T21:52:40.4791860Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212932.xml
2023-01-11T21:52:40.4792223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4792398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4792818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4793017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4793086Z 
2023-01-11T21:52:40.4793191Z Running tests...
2023-01-11T21:52:40.4793458Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4793771Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4794035Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4794253Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7101
2023-01-11T21:52:40.4794468Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7102
2023-01-11T21:52:40.4794831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4794993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4795373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4795568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4795922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4796093Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4796464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4796657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4796894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4797126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4797522Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4797919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4798142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4798369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4798611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.4798845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.4799242Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4799638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4799952Z STAGE:2023-01-11 21:29:43 7102:7102 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4800274Z STAGE:2023-01-11 21:29:43 7101:7101 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4800607Z STAGE:2023-01-11 21:29:43 7101:7101 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4800932Z STAGE:2023-01-11 21:29:43 7102:7102 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4801491Z STAGE:2023-01-11 21:29:43 7101:7101 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:29:43 7102:7102 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4801512Z 
2023-01-11T21:52:40.4801834Z STAGE:2023-01-11 21:29:43 7102:7102 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4802206Z STAGE:2023-01-11 21:29:43 7101:7101 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4802534Z STAGE:2023-01-11 21:29:43 7101:7101 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4802910Z STAGE:2023-01-11 21:29:43 7102:7102 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4803256Z STAGE:2023-01-11 21:29:43 7101:7101 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4803582Z STAGE:2023-01-11 21:29:43 7102:7102 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4803675Z ok (4.231s)
2023-01-11T21:52:40.4803695Z 
2023-01-11T21:52:40.4803955Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4804067Z Ran 1 test in 4.231s
2023-01-11T21:52:40.4804086Z 
2023-01-11T21:52:40.4804176Z OK
2023-01-11T21:52:40.4804406Z 
2023-01-11T21:52:40.4804537Z Generating XML reports...
2023-01-11T21:52:40.4805004Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212939.xml
2023-01-11T21:52:40.4805380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4805542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4805915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4806102Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4806122Z 
2023-01-11T21:52:40.4806230Z Running tests...
2023-01-11T21:52:40.4806485Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4806796Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4807071Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4807281Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7220
2023-01-11T21:52:40.4807499Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7221
2023-01-11T21:52:40.4807854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4808029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4808407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4808588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4808948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4809122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4809489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4809674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4809907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4810306Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4810538Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4810933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4811164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4811385Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4811706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.4811960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.4812419Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4812803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4813117Z STAGE:2023-01-11 21:29:50 7221:7221 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4813445Z STAGE:2023-01-11 21:29:50 7220:7220 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4813991Z STAGE:2023-01-11 21:29:50 7220:7220 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:29:50 7221:7221 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4814014Z 
2023-01-11T21:52:40.4814571Z STAGE:2023-01-11 21:29:50 7221:7221 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:29:50 7220:7220 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4814594Z 
2023-01-11T21:52:40.4814916Z STAGE:2023-01-11 21:29:50 7220:7220 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4815236Z STAGE:2023-01-11 21:29:50 7221:7221 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4815560Z STAGE:2023-01-11 21:29:50 7221:7221 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4815883Z STAGE:2023-01-11 21:29:50 7220:7220 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4816224Z STAGE:2023-01-11 21:29:50 7221:7221 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4816553Z STAGE:2023-01-11 21:29:50 7220:7220 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4816639Z ok (4.218s)
2023-01-11T21:52:40.4816658Z 
2023-01-11T21:52:40.4816920Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4817034Z Ran 1 test in 4.218s
2023-01-11T21:52:40.4817054Z 
2023-01-11T21:52:40.4817144Z OK
2023-01-11T21:52:40.4817163Z 
2023-01-11T21:52:40.4817276Z Generating XML reports...
2023-01-11T21:52:40.4817730Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212946.xml
2023-01-11T21:52:40.4818102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4818270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4818633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4818826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4818846Z 
2023-01-11T21:52:40.4818953Z Running tests...
2023-01-11T21:52:40.4819207Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4819524Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4819795Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4820014Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7339
2023-01-11T21:52:40.4820219Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7340
2023-01-11T21:52:40.4820587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4820744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4821174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4821374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4821783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4821953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4822324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4822512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4822759Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4822986Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4823395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4823784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4824013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4824245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4824477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.4824718Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.4825119Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4825505Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4825835Z STAGE:2023-01-11 21:29:56 7340:7340 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4826148Z STAGE:2023-01-11 21:29:56 7339:7339 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4826487Z STAGE:2023-01-11 21:29:56 7339:7339 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4826804Z STAGE:2023-01-11 21:29:56 7340:7340 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4827145Z STAGE:2023-01-11 21:29:56 7339:7339 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4827487Z STAGE:2023-01-11 21:29:56 7340:7340 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4828002Z STAGE:2023-01-11 21:29:56 7339:7339 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:29:56 7340:7340 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4828022Z 
2023-01-11T21:52:40.4828561Z STAGE:2023-01-11 21:29:56 7339:7339 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:29:56 7340:7340 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4828583Z 
2023-01-11T21:52:40.4828921Z STAGE:2023-01-11 21:29:56 7339:7339 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4829254Z STAGE:2023-01-11 21:29:56 7340:7340 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4829352Z ok (4.241s)
2023-01-11T21:52:40.4829371Z 
2023-01-11T21:52:40.4829619Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4829729Z Ran 1 test in 4.241s
2023-01-11T21:52:40.4829749Z 
2023-01-11T21:52:40.4829839Z OK
2023-01-11T21:52:40.4829858Z 
2023-01-11T21:52:40.4829971Z Generating XML reports...
2023-01-11T21:52:40.4830479Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212953.xml
2023-01-11T21:52:40.4830867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4831082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4831454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4831634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4831654Z 
2023-01-11T21:52:40.4831744Z Running tests...
2023-01-11T21:52:40.4831994Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4832300Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4832603Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4832817Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7458
2023-01-11T21:52:40.4833025Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7459
2023-01-11T21:52:40.4833393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4833563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4833923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4834111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4834474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4834648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4835011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4835197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4835444Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4835683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4836085Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4836467Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4836696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4836917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4837157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.4837404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.4837794Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4838188Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.4838517Z STAGE:2023-01-11 21:30:03 7458:7458 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4838835Z STAGE:2023-01-11 21:30:03 7459:7459 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4839359Z STAGE:2023-01-11 21:30:03 7458:7458 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:30:03 7459:7459 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4839392Z 
2023-01-11T21:52:40.4839993Z STAGE:2023-01-11 21:30:03 7459:7459 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:30:03 7458:7458 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4840034Z 
2023-01-11T21:52:40.4840398Z STAGE:2023-01-11 21:30:03 7458:7458 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4840708Z STAGE:2023-01-11 21:30:03 7459:7459 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4841034Z STAGE:2023-01-11 21:30:03 7458:7458 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4841356Z STAGE:2023-01-11 21:30:03 7459:7459 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4841689Z STAGE:2023-01-11 21:30:03 7458:7458 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4842026Z STAGE:2023-01-11 21:30:03 7459:7459 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4842127Z ok (4.235s)
2023-01-11T21:52:40.4842151Z 
2023-01-11T21:52:40.4842414Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4842508Z Ran 1 test in 4.235s
2023-01-11T21:52:40.4842531Z 
2023-01-11T21:52:40.4842613Z OK
2023-01-11T21:52:40.4842632Z 
2023-01-11T21:52:40.4842751Z Generating XML reports...
2023-01-11T21:52:40.4843209Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212959.xml
2023-01-11T21:52:40.4843571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4843745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4844118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4844530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4844551Z 
2023-01-11T21:52:40.4844665Z Running tests...
2023-01-11T21:52:40.4844924Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4845244Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4845505Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4845719Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7577
2023-01-11T21:52:40.4845936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7578
2023-01-11T21:52:40.4846303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4846476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4846852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4847032Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4847403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4847570Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4847939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4848129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4848377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4848613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4849012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4849518Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4849745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4850028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4850190Z skip: Skipped due to small world size. (4.139s)
2023-01-11T21:52:40.4850210Z 
2023-01-11T21:52:40.4850469Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4850577Z Ran 1 test in 4.139s
2023-01-11T21:52:40.4850597Z 
2023-01-11T21:52:40.4850704Z OK (skipped=1)
2023-01-11T21:52:40.4850724Z 
2023-01-11T21:52:40.4850838Z Generating XML reports...
2023-01-11T21:52:40.4851295Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213006.xml
2023-01-11T21:52:40.4851654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4851835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4852210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4852403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4852422Z 
2023-01-11T21:52:40.4852531Z Running tests...
2023-01-11T21:52:40.4852783Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4853097Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4853362Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4853572Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7686
2023-01-11T21:52:40.4853770Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7687
2023-01-11T21:52:40.4854142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4854316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4854694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4854880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4855239Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4855412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4855774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4855943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4856190Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4856591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4856829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4857218Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4857448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4857668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4857823Z skip: Skipped due to small world size. (4.099s)
2023-01-11T21:52:40.4857843Z 
2023-01-11T21:52:40.4858099Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4858193Z Ran 1 test in 4.099s
2023-01-11T21:52:40.4858227Z 
2023-01-11T21:52:40.4858368Z OK (skipped=1)
2023-01-11T21:52:40.4858389Z 
2023-01-11T21:52:40.4858515Z Generating XML reports...
2023-01-11T21:52:40.4858976Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213013.xml
2023-01-11T21:52:40.4859389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4859565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4859939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4860127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4860147Z 
2023-01-11T21:52:40.4860255Z Running tests...
2023-01-11T21:52:40.4860503Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4860811Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4861078Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4861299Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7795
2023-01-11T21:52:40.4861514Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7796
2023-01-11T21:52:40.4861877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4862051Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4862423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4862595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4862960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4863134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4863506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4863688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4863932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4864333Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4864566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4864963Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4865174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4865399Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4865552Z skip: Skipped due to small world size. (4.149s)
2023-01-11T21:52:40.4865574Z 
2023-01-11T21:52:40.4865839Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4865951Z Ran 1 test in 4.149s
2023-01-11T21:52:40.4865970Z 
2023-01-11T21:52:40.4866069Z OK (skipped=1)
2023-01-11T21:52:40.4866087Z 
2023-01-11T21:52:40.4866207Z Generating XML reports...
2023-01-11T21:52:40.4866654Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213019.xml
2023-01-11T21:52:40.4867023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4867183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4867614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4867803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4867865Z 
2023-01-11T21:52:40.4867973Z Running tests...
2023-01-11T21:52:40.4868237Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4868540Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4868803Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4869019Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 7904
2023-01-11T21:52:40.4869217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 7905
2023-01-11T21:52:40.4869579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4869753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4870122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4870314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4870671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4870841Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4871219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4871398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4871629Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4871869Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4872275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4872664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4872896Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4873116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4873272Z skip: Skipped due to small world size. (4.156s)
2023-01-11T21:52:40.4873292Z 
2023-01-11T21:52:40.4873557Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4873652Z Ran 1 test in 4.156s
2023-01-11T21:52:40.4873678Z 
2023-01-11T21:52:40.4873768Z OK (skipped=1)
2023-01-11T21:52:40.4873787Z 
2023-01-11T21:52:40.4873905Z Generating XML reports...
2023-01-11T21:52:40.4874352Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213026.xml
2023-01-11T21:52:40.4874726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4874905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4875277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4875466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4875486Z 
2023-01-11T21:52:40.4875582Z Running tests...
2023-01-11T21:52:40.4875827Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4876140Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4876396Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4876654Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8013
2023-01-11T21:52:40.4876876Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8014
2023-01-11T21:52:40.4877293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4877470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4877844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4878031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4878381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4878559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4878930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4879116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4879361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4879598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4880001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4880399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4880613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4880832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4881168Z STAGE:2023-01-11 21:30:37 8014:8014 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4881497Z STAGE:2023-01-11 21:30:37 8013:8013 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4882043Z STAGE:2023-01-11 21:30:37 8013:8013 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:30:37 8014:8014 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4882064Z 
2023-01-11T21:52:40.4882618Z STAGE:2023-01-11 21:30:37 8013:8013 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:30:37 8014:8014 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4882638Z 
2023-01-11T21:52:40.4882963Z STAGE:2023-01-11 21:30:37 8013:8013 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4883282Z STAGE:2023-01-11 21:30:37 8014:8014 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4883605Z STAGE:2023-01-11 21:30:37 8013:8013 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4883926Z STAGE:2023-01-11 21:30:37 8014:8014 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4884571Z STAGE:2023-01-11 21:30:37 8013:8013 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4884918Z STAGE:2023-01-11 21:30:37 8014:8014 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4885011Z ok (4.250s)
2023-01-11T21:52:40.4885031Z 
2023-01-11T21:52:40.4885291Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4885401Z Ran 1 test in 4.250s
2023-01-11T21:52:40.4885421Z 
2023-01-11T21:52:40.4885505Z OK
2023-01-11T21:52:40.4885525Z 
2023-01-11T21:52:40.4885644Z Generating XML reports...
2023-01-11T21:52:40.4886100Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213033.xml
2023-01-11T21:52:40.4886537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4886706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4887148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4887340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4887360Z 
2023-01-11T21:52:40.4887464Z Running tests...
2023-01-11T21:52:40.4887722Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4888039Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4888296Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4888507Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8126
2023-01-11T21:52:40.4888723Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8127
2023-01-11T21:52:40.4889082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4889253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4889631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4889820Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4890175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4890346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4890710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4890901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4891131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4891376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4891774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4892171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4892405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4892625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4892956Z STAGE:2023-01-11 21:30:43 8126:8126 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4893289Z STAGE:2023-01-11 21:30:43 8127:8127 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4893618Z STAGE:2023-01-11 21:30:43 8126:8126 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4893939Z STAGE:2023-01-11 21:30:43 8127:8127 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4894503Z STAGE:2023-01-11 21:30:43 8127:8127 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:30:43 8126:8126 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4894524Z 
2023-01-11T21:52:40.4894850Z STAGE:2023-01-11 21:30:43 8126:8126 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4895163Z STAGE:2023-01-11 21:30:43 8127:8127 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4895488Z STAGE:2023-01-11 21:30:43 8127:8127 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4895858Z STAGE:2023-01-11 21:30:43 8126:8126 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4896212Z STAGE:2023-01-11 21:30:43 8127:8127 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4896600Z STAGE:2023-01-11 21:30:43 8126:8126 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4896699Z ok (4.310s)
2023-01-11T21:52:40.4896719Z 
2023-01-11T21:52:40.4896980Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4897076Z Ran 1 test in 4.311s
2023-01-11T21:52:40.4897096Z 
2023-01-11T21:52:40.4897178Z OK
2023-01-11T21:52:40.4897197Z 
2023-01-11T21:52:40.4897315Z Generating XML reports...
2023-01-11T21:52:40.4897773Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213040.xml
2023-01-11T21:52:40.4898140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4898319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4898703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4898889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4898909Z 
2023-01-11T21:52:40.4898999Z Running tests...
2023-01-11T21:52:40.4899259Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4899577Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4899832Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4900047Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8239
2023-01-11T21:52:40.4900265Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8240
2023-01-11T21:52:40.4900633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4900804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4901174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4901364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4901729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4901891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4902261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4902448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4902700Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4902939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4903338Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4903723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4903954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4904174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4904504Z STAGE:2023-01-11 21:30:51 8240:8240 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4905333Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1582: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T21:52:40.4905454Z   warnings.warn(
2023-01-11T21:52:40.4905832Z STAGE:2023-01-11 21:30:51 8239:8239 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4906591Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1582: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T21:52:40.4906703Z   warnings.warn(
2023-01-11T21:52:40.4907025Z STAGE:2023-01-11 21:30:51 8239:8239 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4907333Z STAGE:2023-01-11 21:30:51 8240:8240 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4907901Z STAGE:2023-01-11 21:30:51 8240:8240 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:30:51 8239:8239 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4907921Z 
2023-01-11T21:52:40.4908450Z STAGE:2023-01-11 21:30:51 8239:8239 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:30:51 8240:8240 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4908470Z 
2023-01-11T21:52:40.4908802Z STAGE:2023-01-11 21:30:51 8239:8239 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4909117Z STAGE:2023-01-11 21:30:51 8240:8240 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4909456Z STAGE:2023-01-11 21:30:51 8239:8239 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4909797Z STAGE:2023-01-11 21:30:51 8240:8240 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4909888Z ok (5.316s)
2023-01-11T21:52:40.4909908Z 
2023-01-11T21:52:40.4910172Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4910283Z Ran 1 test in 5.316s
2023-01-11T21:52:40.4910302Z 
2023-01-11T21:52:40.4910380Z OK
2023-01-11T21:52:40.4910416Z 
2023-01-11T21:52:40.4910520Z Generating XML reports...
2023-01-11T21:52:40.4910967Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213046.xml
2023-01-11T21:52:40.4911339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4911515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4911899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4912085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4912104Z 
2023-01-11T21:52:40.4912208Z Running tests...
2023-01-11T21:52:40.4912474Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4912773Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4913046Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4913260Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8354
2023-01-11T21:52:40.4913477Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8355
2023-01-11T21:52:40.4913840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4914014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4914394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4914626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4914984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4915197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4915576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4915794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4916041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4916290Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4916695Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4917085Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4917333Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4917550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4917881Z STAGE:2023-01-11 21:30:59 8354:8354 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4918641Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1582: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T21:52:40.4918753Z   warnings.warn(
2023-01-11T21:52:40.4919072Z STAGE:2023-01-11 21:30:59 8355:8355 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4919840Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1582: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T21:52:40.4919956Z   warnings.warn(
2023-01-11T21:52:40.4920498Z STAGE:2023-01-11 21:30:59 8354:8354 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:30:59 8355:8355 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4920519Z 
2023-01-11T21:52:40.4921071Z STAGE:2023-01-11 21:30:59 8354:8354 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:30:59 8355:8355 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4921091Z 
2023-01-11T21:52:40.4921611Z STAGE:2023-01-11 21:30:59 8355:8355 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:30:59 8354:8354 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4921634Z 
2023-01-11T21:52:40.4921967Z STAGE:2023-01-11 21:30:59 8354:8354 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4922288Z STAGE:2023-01-11 21:30:59 8355:8355 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4922621Z STAGE:2023-01-11 21:30:59 8354:8354 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4922963Z STAGE:2023-01-11 21:30:59 8355:8355 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4923066Z ok (5.296s)
2023-01-11T21:52:40.4923086Z 
2023-01-11T21:52:40.4923344Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4923452Z Ran 1 test in 5.296s
2023-01-11T21:52:40.4923472Z 
2023-01-11T21:52:40.4923564Z OK
2023-01-11T21:52:40.4923583Z 
2023-01-11T21:52:40.4923698Z Generating XML reports...
2023-01-11T21:52:40.4924385Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213054.xml
2023-01-11T21:52:40.4924795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4925032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4925410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4925601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4925621Z 
2023-01-11T21:52:40.4925729Z Running tests...
2023-01-11T21:52:40.4925984Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4926298Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4926562Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4926777Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8469
2023-01-11T21:52:40.4926978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8470
2023-01-11T21:52:40.4927350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4927529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4927913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4928095Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4928459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4928624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4928998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4929187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4929419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4929667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4930063Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4930458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4930682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4930909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4931242Z STAGE:2023-01-11 21:31:06 8469:8469 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4931573Z STAGE:2023-01-11 21:31:06 8470:8470 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4932104Z STAGE:2023-01-11 21:31:06 8470:8470 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:31:06 8469:8469 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4932129Z 
2023-01-11T21:52:40.4932736Z STAGE:2023-01-11 21:31:06 8469:8469 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:31:06 8470:8470 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4932758Z 
2023-01-11T21:52:40.4933069Z STAGE:2023-01-11 21:31:06 8469:8469 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4933382Z STAGE:2023-01-11 21:31:06 8470:8470 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4933772Z STAGE:2023-01-11 21:31:06 8470:8470 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4934108Z STAGE:2023-01-11 21:31:06 8469:8469 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4934441Z STAGE:2023-01-11 21:31:06 8470:8470 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4934825Z STAGE:2023-01-11 21:31:06 8469:8469 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4934929Z ok (4.340s)
2023-01-11T21:52:40.4934948Z 
2023-01-11T21:52:40.4935202Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4935309Z Ran 1 test in 4.340s
2023-01-11T21:52:40.4935329Z 
2023-01-11T21:52:40.4935403Z OK
2023-01-11T21:52:40.4935421Z 
2023-01-11T21:52:40.4935548Z Generating XML reports...
2023-01-11T21:52:40.4936004Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213102.xml
2023-01-11T21:52:40.4936376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4936552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4936928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4937117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4937136Z 
2023-01-11T21:52:40.4937243Z Running tests...
2023-01-11T21:52:40.4937488Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4937809Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4938067Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4938284Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8582
2023-01-11T21:52:40.4938496Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8583
2023-01-11T21:52:40.4938870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4939049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4939431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4939613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4939963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4940131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4940497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4940683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4940933Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4941169Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4941569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4941966Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4942187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4942401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4942499Z ok (5.058s)
2023-01-11T21:52:40.4942519Z 
2023-01-11T21:52:40.4942783Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4942887Z Ran 1 test in 5.058s
2023-01-11T21:52:40.4942960Z 
2023-01-11T21:52:40.4943057Z OK
2023-01-11T21:52:40.4943075Z 
2023-01-11T21:52:40.4943201Z Generating XML reports...
2023-01-11T21:52:40.4943652Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213109.xml
2023-01-11T21:52:40.4944076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4944239Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4944621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4944803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4944823Z 
2023-01-11T21:52:40.4944926Z Running tests...
2023-01-11T21:52:40.4945191Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4945502Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4945754Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4945977Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8693
2023-01-11T21:52:40.4946175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8694
2023-01-11T21:52:40.4946542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4946715Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4947095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4947282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4947645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4947821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4948190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4948381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4948611Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4948853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4949257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4949649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4949879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4950107Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4950427Z STAGE:2023-01-11 21:31:20 8694:8694 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4950756Z STAGE:2023-01-11 21:31:20 8693:8693 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4951298Z STAGE:2023-01-11 21:31:20 8694:8694 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:31:20 8693:8693 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4951319Z 
2023-01-11T21:52:40.4951874Z STAGE:2023-01-11 21:31:20 8693:8693 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:31:20 8694:8694 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4951894Z 
2023-01-11T21:52:40.4952201Z STAGE:2023-01-11 21:31:20 8694:8694 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4952573Z STAGE:2023-01-11 21:31:20 8693:8693 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4953124Z STAGE:2023-01-11 21:31:20 8694:8694 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:31:20 8693:8693 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4953185Z 
2023-01-11T21:52:40.4953748Z STAGE:2023-01-11 21:31:20 8694:8694 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:31:20 8693:8693 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4953767Z 
2023-01-11T21:52:40.4953864Z ok (4.240s)
2023-01-11T21:52:40.4953883Z 
2023-01-11T21:52:40.4954146Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4954251Z Ran 1 test in 4.241s
2023-01-11T21:52:40.4954270Z 
2023-01-11T21:52:40.4954359Z OK
2023-01-11T21:52:40.4954378Z 
2023-01-11T21:52:40.4954500Z Generating XML reports...
2023-01-11T21:52:40.4954950Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213117.xml
2023-01-11T21:52:40.4955327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4955490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4955874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4956057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4956077Z 
2023-01-11T21:52:40.4956179Z Running tests...
2023-01-11T21:52:40.4956439Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4956756Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4957024Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4957243Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8806
2023-01-11T21:52:40.4957445Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8807
2023-01-11T21:52:40.4957817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4957993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4958373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4958566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4958933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4959109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4959479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4959664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4959898Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4960144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4960546Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4960945Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4961167Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4961395Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4961777Z STAGE:2023-01-11 21:31:27 8806:8806 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4962119Z STAGE:2023-01-11 21:31:27 8807:8807 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4962484Z STAGE:2023-01-11 21:31:27 8806:8806 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4962817Z STAGE:2023-01-11 21:31:27 8807:8807 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4963159Z STAGE:2023-01-11 21:31:27 8806:8806 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4963492Z STAGE:2023-01-11 21:31:27 8807:8807 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4963809Z STAGE:2023-01-11 21:31:27 8806:8806 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4964135Z STAGE:2023-01-11 21:31:27 8807:8807 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4964699Z STAGE:2023-01-11 21:31:27 8807:8807 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4965027Z STAGE:2023-01-11 21:31:27 8806:8806 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4965596Z STAGE:2023-01-11 21:31:27 8806:8806 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:31:27 8807:8807 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4965617Z 
2023-01-11T21:52:40.4965708Z ok (4.218s)
2023-01-11T21:52:40.4965728Z 
2023-01-11T21:52:40.4965974Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4966083Z Ran 1 test in 4.218s
2023-01-11T21:52:40.4966102Z 
2023-01-11T21:52:40.4966193Z OK
2023-01-11T21:52:40.4966212Z 
2023-01-11T21:52:40.4966335Z Generating XML reports...
2023-01-11T21:52:40.4966790Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213123.xml
2023-01-11T21:52:40.4967161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4967342Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4967712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4967889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4967923Z 
2023-01-11T21:52:40.4968014Z Running tests...
2023-01-11T21:52:40.4968274Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4968584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4968855Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4969077Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 8919
2023-01-11T21:52:40.4969294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 8920
2023-01-11T21:52:40.4969670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4969850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4970216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4970410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4970775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4970948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4971323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4971590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4971851Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4972166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4972575Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4972958Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4973189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4973416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4973747Z STAGE:2023-01-11 21:31:34 8920:8920 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4974083Z STAGE:2023-01-11 21:31:34 8919:8919 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4974416Z STAGE:2023-01-11 21:31:34 8919:8919 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4974745Z STAGE:2023-01-11 21:31:34 8920:8920 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4975088Z STAGE:2023-01-11 21:31:34 8919:8919 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4975407Z STAGE:2023-01-11 21:31:34 8920:8920 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4975728Z STAGE:2023-01-11 21:31:34 8919:8919 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4976052Z STAGE:2023-01-11 21:31:34 8920:8920 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4976386Z STAGE:2023-01-11 21:31:34 8919:8919 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4976936Z STAGE:2023-01-11 21:31:34 8920:8920 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:31:34 8919:8919 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4976959Z 
2023-01-11T21:52:40.4977297Z STAGE:2023-01-11 21:31:34 8920:8920 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4977398Z ok (4.219s)
2023-01-11T21:52:40.4977417Z 
2023-01-11T21:52:40.4977679Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4977789Z Ran 1 test in 4.220s
2023-01-11T21:52:40.4977809Z 
2023-01-11T21:52:40.4977901Z OK
2023-01-11T21:52:40.4977920Z 
2023-01-11T21:52:40.4978026Z Generating XML reports...
2023-01-11T21:52:40.4978479Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213130.xml
2023-01-11T21:52:40.4978856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4979034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4979414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4979608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4979628Z 
2023-01-11T21:52:40.4979740Z Running tests...
2023-01-11T21:52:40.4980001Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4980301Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4980563Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4980781Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9032
2023-01-11T21:52:40.4981045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9033
2023-01-11T21:52:40.4981429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4981651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4982038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4982229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4982596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4982751Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4983127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4983314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4983566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4983809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4984215Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4984613Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4984848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4985077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4985392Z STAGE:2023-01-11 21:31:42 9033:9033 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4985721Z STAGE:2023-01-11 21:31:42 9032:9032 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4986052Z STAGE:2023-01-11 21:31:42 9032:9032 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4986384Z STAGE:2023-01-11 21:31:42 9033:9033 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4986733Z STAGE:2023-01-11 21:31:42 9032:9032 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4987073Z STAGE:2023-01-11 21:31:42 9033:9033 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4987394Z STAGE:2023-01-11 21:31:42 9033:9033 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4987715Z STAGE:2023-01-11 21:31:42 9032:9032 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4988254Z STAGE:2023-01-11 21:31:43 9033:9033 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:31:43 9032:9032 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4988278Z 
2023-01-11T21:52:40.4988826Z STAGE:2023-01-11 21:31:43 9032:9032 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:31:43 9033:9033 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4988866Z 
2023-01-11T21:52:40.4989171Z STAGE:2023-01-11 21:31:43 9033:9033 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4989489Z STAGE:2023-01-11 21:31:43 9032:9032 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4990025Z STAGE:2023-01-11 21:31:43 9032:9032 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:31:43 9033:9033 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4990045Z 
2023-01-11T21:52:40.4990382Z STAGE:2023-01-11 21:31:43 9032:9032 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4990773Z STAGE:2023-01-11 21:31:43 9033:9033 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.4990881Z ok (6.163s)
2023-01-11T21:52:40.4990901Z 
2023-01-11T21:52:40.4991169Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4991367Z Ran 1 test in 6.163s
2023-01-11T21:52:40.4991387Z 
2023-01-11T21:52:40.4991478Z OK
2023-01-11T21:52:40.4991498Z 
2023-01-11T21:52:40.4991602Z Generating XML reports...
2023-01-11T21:52:40.4992060Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213137.xml
2023-01-11T21:52:40.4992435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4992614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4992994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4993191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4993210Z 
2023-01-11T21:52:40.4993320Z Running tests...
2023-01-11T21:52:40.4993582Z ----------------------------------------------------------------------
2023-01-11T21:52:40.4993901Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.4994157Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.4994376Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9148
2023-01-11T21:52:40.4994594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9149
2023-01-11T21:52:40.4994965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4995140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4995525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4995717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4996082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.4996241Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.4996619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.4996808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.4997054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.4997297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.4997699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4998096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.4998332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.4998559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.4998876Z STAGE:2023-01-11 21:31:50 9148:9148 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4999205Z STAGE:2023-01-11 21:31:50 9149:9149 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.4999541Z STAGE:2023-01-11 21:31:50 9148:9148 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.4999871Z STAGE:2023-01-11 21:31:50 9149:9149 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5000488Z STAGE:2023-01-11 21:31:50 9148:9148 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:31:50 9149:9149 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5000510Z 
2023-01-11T21:52:40.5000878Z STAGE:2023-01-11 21:31:50 9148:9148 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5001200Z STAGE:2023-01-11 21:31:50 9149:9149 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5001528Z STAGE:2023-01-11 21:31:51 9148:9148 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5001850Z STAGE:2023-01-11 21:31:51 9149:9149 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5002191Z STAGE:2023-01-11 21:31:51 9148:9148 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5002516Z STAGE:2023-01-11 21:31:51 9149:9149 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5002840Z STAGE:2023-01-11 21:31:51 9149:9149 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5003162Z STAGE:2023-01-11 21:31:51 9148:9148 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5003494Z STAGE:2023-01-11 21:31:51 9149:9149 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5003819Z STAGE:2023-01-11 21:31:51 9148:9148 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5004160Z STAGE:2023-01-11 21:31:51 9149:9149 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5004742Z STAGE:2023-01-11 21:31:51 9148:9148 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5004849Z ok (6.092s)
2023-01-11T21:52:40.5004869Z 
2023-01-11T21:52:40.5005131Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5005226Z Ran 1 test in 6.092s
2023-01-11T21:52:40.5005245Z 
2023-01-11T21:52:40.5005338Z OK
2023-01-11T21:52:40.5005357Z 
2023-01-11T21:52:40.5005481Z Generating XML reports...
2023-01-11T21:52:40.5005936Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213146.xml
2023-01-11T21:52:40.5006319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5006497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5006882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5007077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5007096Z 
2023-01-11T21:52:40.5007186Z Running tests...
2023-01-11T21:52:40.5007448Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5007763Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5008041Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5008259Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 9264
2023-01-11T21:52:40.5008482Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 9265
2023-01-11T21:52:40.5008856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5009032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5009395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5009588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5009956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5010206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5010596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5010847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5011097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5011346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5011752Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5012132Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5012365Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5012598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5012932Z STAGE:2023-01-11 21:31:59 9265:9265 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5013264Z STAGE:2023-01-11 21:31:59 9264:9264 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5013597Z STAGE:2023-01-11 21:31:59 9264:9264 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5013932Z STAGE:2023-01-11 21:31:59 9265:9265 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5014276Z STAGE:2023-01-11 21:31:59 9264:9264 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5014618Z STAGE:2023-01-11 21:31:59 9265:9265 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5014921Z STAGE:2023-01-11 21:31:59 9265:9265 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5015245Z STAGE:2023-01-11 21:31:59 9264:9264 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5015784Z STAGE:2023-01-11 21:32:00 9264:9264 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:32:00 9265:9265 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5015808Z 
2023-01-11T21:52:40.5016371Z STAGE:2023-01-11 21:32:00 9265:9265 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:32:00 9264:9264 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5016391Z 
2023-01-11T21:52:40.5016715Z STAGE:2023-01-11 21:32:00 9265:9265 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5017035Z STAGE:2023-01-11 21:32:00 9264:9264 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5017571Z STAGE:2023-01-11 21:32:00 9264:9264 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:32:00 9265:9265 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5017591Z 
2023-01-11T21:52:40.5018155Z STAGE:2023-01-11 21:32:00 9264:9264 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:32:00 9265:9265 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5018178Z 
2023-01-11T21:52:40.5018279Z ok (6.135s)
2023-01-11T21:52:40.5018299Z 
2023-01-11T21:52:40.5018565Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5018676Z Ran 1 test in 6.135s
2023-01-11T21:52:40.5018695Z 
2023-01-11T21:52:40.5018785Z OK
2023-01-11T21:52:40.5018804Z 
2023-01-11T21:52:40.5018910Z Generating XML reports...
2023-01-11T21:52:40.5019368Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213154.xml
2023-01-11T21:52:40.5019744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5019973Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5020372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5020608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5020627Z 
2023-01-11T21:52:40.5020735Z Running tests...
2023-01-11T21:52:40.5020997Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5021298Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5021545Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T21:52:40.5021564Z 
2023-01-11T21:52:40.5021824Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5021935Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5021954Z 
2023-01-11T21:52:40.5022060Z OK (skipped=1)
2023-01-11T21:52:40.5022083Z 
2023-01-11T21:52:40.5022205Z Generating XML reports...
2023-01-11T21:52:40.5022654Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213203.xml
2023-01-11T21:52:40.5023031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5023208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5023573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5023763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5023782Z 
2023-01-11T21:52:40.5023889Z Running tests...
2023-01-11T21:52:40.5024150Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5024464Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5024726Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T21:52:40.5024746Z 
2023-01-11T21:52:40.5025009Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5025122Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5025142Z 
2023-01-11T21:52:40.5025247Z OK (skipped=1)
2023-01-11T21:52:40.5025266Z 
2023-01-11T21:52:40.5025371Z Generating XML reports...
2023-01-11T21:52:40.5025824Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213205.xml
2023-01-11T21:52:40.5026197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5026373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5026756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5026950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5026970Z 
2023-01-11T21:52:40.5027077Z Running tests...
2023-01-11T21:52:40.5027344Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5027642Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5027902Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2023-01-11T21:52:40.5027922Z 
2023-01-11T21:52:40.5028178Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5028288Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5028307Z 
2023-01-11T21:52:40.5028412Z OK (skipped=1)
2023-01-11T21:52:40.5028431Z 
2023-01-11T21:52:40.5028553Z Generating XML reports...
2023-01-11T21:52:40.5029002Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213208.xml
2023-01-11T21:52:40.5029422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5029606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5030017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5030209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5030228Z 
2023-01-11T21:52:40.5030336Z Running tests...
2023-01-11T21:52:40.5030597Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5030914Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5031184Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2023-01-11T21:52:40.5031204Z 
2023-01-11T21:52:40.5031465Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5031576Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5034255Z 
2023-01-11T21:52:40.5034390Z OK (skipped=1)
2023-01-11T21:52:40.5034416Z 
2023-01-11T21:52:40.5034522Z Generating XML reports...
2023-01-11T21:52:40.5034984Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213210.xml
2023-01-11T21:52:40.5035356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5035532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5035914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5036105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5036124Z 
2023-01-11T21:52:40.5036232Z Running tests...
2023-01-11T21:52:40.5036495Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5036811Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5037057Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T21:52:40.5037094Z 
2023-01-11T21:52:40.5037333Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5037444Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5037463Z 
2023-01-11T21:52:40.5037570Z OK (skipped=1)
2023-01-11T21:52:40.5037588Z 
2023-01-11T21:52:40.5037712Z Generating XML reports...
2023-01-11T21:52:40.5038161Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213212.xml
2023-01-11T21:52:40.5038535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5038716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5039097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5039275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5039295Z 
2023-01-11T21:52:40.5039403Z Running tests...
2023-01-11T21:52:40.5039666Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5039986Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5040260Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2023-01-11T21:52:40.5040280Z 
2023-01-11T21:52:40.5040537Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5040649Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5040668Z 
2023-01-11T21:52:40.5040773Z OK (skipped=1)
2023-01-11T21:52:40.5040792Z 
2023-01-11T21:52:40.5040979Z Generating XML reports...
2023-01-11T21:52:40.5041425Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213215.xml
2023-01-11T21:52:40.5041862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5042037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5042416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5042607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5042626Z 
2023-01-11T21:52:40.5042736Z Running tests...
2023-01-11T21:52:40.5042999Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5043314Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5043556Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T21:52:40.5043596Z 
2023-01-11T21:52:40.5043839Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5043953Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5043972Z 
2023-01-11T21:52:40.5044081Z OK (skipped=1)
2023-01-11T21:52:40.5044100Z 
2023-01-11T21:52:40.5044507Z Generating XML reports...
2023-01-11T21:52:40.5044982Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213217.xml
2023-01-11T21:52:40.5045356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5045531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5045909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5046089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5046108Z 
2023-01-11T21:52:40.5046215Z Running tests...
2023-01-11T21:52:40.5046480Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5046804Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5047082Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T21:52:40.5047101Z 
2023-01-11T21:52:40.5047361Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5047471Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5047490Z 
2023-01-11T21:52:40.5047596Z OK (skipped=1)
2023-01-11T21:52:40.5047615Z 
2023-01-11T21:52:40.5047738Z Generating XML reports...
2023-01-11T21:52:40.5048170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213220.xml
2023-01-11T21:52:40.5048542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5048718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5049104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5049293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5049313Z 
2023-01-11T21:52:40.5049424Z Running tests...
2023-01-11T21:52:40.5049686Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5050000Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5050268Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:52:40.5050306Z 
2023-01-11T21:52:40.5050628Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5050749Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5050768Z 
2023-01-11T21:52:40.5050876Z OK (skipped=1)
2023-01-11T21:52:40.5050895Z 
2023-01-11T21:52:40.5051069Z Generating XML reports...
2023-01-11T21:52:40.5051520Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213222.xml
2023-01-11T21:52:40.5051890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5052066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5052440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5052615Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5052653Z 
2023-01-11T21:52:40.5052743Z Running tests...
2023-01-11T21:52:40.5053008Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5053324Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5053622Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:52:40.5053645Z 
2023-01-11T21:52:40.5053900Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5054011Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5054030Z 
2023-01-11T21:52:40.5054135Z OK (skipped=1)
2023-01-11T21:52:40.5054154Z 
2023-01-11T21:52:40.5054275Z Generating XML reports...
2023-01-11T21:52:40.5054709Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213224.xml
2023-01-11T21:52:40.5055080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5055260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5055637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5055835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5055854Z 
2023-01-11T21:52:40.5055961Z Running tests...
2023-01-11T21:52:40.5056223Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5056536Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5056829Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T21:52:40.5056848Z 
2023-01-11T21:52:40.5057087Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5057197Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5057216Z 
2023-01-11T21:52:40.5057323Z OK (skipped=1)
2023-01-11T21:52:40.5057341Z 
2023-01-11T21:52:40.5057465Z Generating XML reports...
2023-01-11T21:52:40.5057916Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213227.xml
2023-01-11T21:52:40.5058292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5058469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5058852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5059044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5059064Z 
2023-01-11T21:52:40.5059154Z Running tests...
2023-01-11T21:52:40.5059414Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5059727Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5060082Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T21:52:40.5060104Z 
2023-01-11T21:52:40.5060406Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5060516Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5060536Z 
2023-01-11T21:52:40.5060643Z OK (skipped=1)
2023-01-11T21:52:40.5060662Z 
2023-01-11T21:52:40.5060783Z Generating XML reports...
2023-01-11T21:52:40.5061213Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213229.xml
2023-01-11T21:52:40.5061585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5061760Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5062139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5062335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5062355Z 
2023-01-11T21:52:40.5062461Z Running tests...
2023-01-11T21:52:40.5062727Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5063046Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5063348Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:52:40.5063368Z 
2023-01-11T21:52:40.5063610Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5063722Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5063741Z 
2023-01-11T21:52:40.5063847Z OK (skipped=1)
2023-01-11T21:52:40.5063866Z 
2023-01-11T21:52:40.5063989Z Generating XML reports...
2023-01-11T21:52:40.5064445Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213231.xml
2023-01-11T21:52:40.5064819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5064998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5065381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5065577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5065597Z 
2023-01-11T21:52:40.5065688Z Running tests...
2023-01-11T21:52:40.5065952Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5066265Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5066573Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T21:52:40.5066597Z 
2023-01-11T21:52:40.5066859Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5066972Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5066994Z 
2023-01-11T21:52:40.5067101Z OK (skipped=1)
2023-01-11T21:52:40.5067120Z 
2023-01-11T21:52:40.5067243Z Generating XML reports...
2023-01-11T21:52:40.5067691Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213234.xml
2023-01-11T21:52:40.5068045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5068223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5068604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5068796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5068815Z 
2023-01-11T21:52:40.5068977Z Running tests...
2023-01-11T21:52:40.5069249Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5069565Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5069902Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:52:40.5069922Z 
2023-01-11T21:52:40.5070184Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5070277Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5070296Z 
2023-01-11T21:52:40.5070401Z OK (skipped=1)
2023-01-11T21:52:40.5070420Z 
2023-01-11T21:52:40.5070540Z Generating XML reports...
2023-01-11T21:52:40.5070992Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213236.xml
2023-01-11T21:52:40.5071365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5071541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5071923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5072120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5072139Z 
2023-01-11T21:52:40.5072248Z Running tests...
2023-01-11T21:52:40.5072494Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5072812Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5073117Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T21:52:40.5073136Z 
2023-01-11T21:52:40.5073393Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5073504Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5073527Z 
2023-01-11T21:52:40.5073635Z OK (skipped=1)
2023-01-11T21:52:40.5073654Z 
2023-01-11T21:52:40.5073778Z Generating XML reports...
2023-01-11T21:52:40.5074228Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213239.xml
2023-01-11T21:52:40.5074584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5074757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5075138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5075328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5075347Z 
2023-01-11T21:52:40.5075454Z Running tests...
2023-01-11T21:52:40.5075715Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5076031Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5076319Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:52:40.5076342Z 
2023-01-11T21:52:40.5076600Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5076693Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5076712Z 
2023-01-11T21:52:40.5076818Z OK (skipped=1)
2023-01-11T21:52:40.5076836Z 
2023-01-11T21:52:40.5076957Z Generating XML reports...
2023-01-11T21:52:40.5077410Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213241.xml
2023-01-11T21:52:40.5077784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5077959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5078385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5078585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5078643Z 
2023-01-11T21:52:40.5078755Z Running tests...
2023-01-11T21:52:40.5079002Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5079319Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5079618Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:52:40.5079638Z 
2023-01-11T21:52:40.5079897Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5080008Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5080027Z 
2023-01-11T21:52:40.5080133Z OK (skipped=1)
2023-01-11T21:52:40.5080152Z 
2023-01-11T21:52:40.5080275Z Generating XML reports...
2023-01-11T21:52:40.5080727Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213243.xml
2023-01-11T21:52:40.5081101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5081264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5081646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5081837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5081856Z 
2023-01-11T21:52:40.5081964Z Running tests...
2023-01-11T21:52:40.5082223Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5082534Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5082833Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T21:52:40.5082853Z 
2023-01-11T21:52:40.5083112Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5083208Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5083245Z 
2023-01-11T21:52:40.5083333Z OK (skipped=1)
2023-01-11T21:52:40.5083352Z 
2023-01-11T21:52:40.5083473Z Generating XML reports...
2023-01-11T21:52:40.5083919Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213246.xml
2023-01-11T21:52:40.5084519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5084709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5085098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5085295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5085314Z 
2023-01-11T21:52:40.5085421Z Running tests...
2023-01-11T21:52:40.5085665Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5085984Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5086292Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T21:52:40.5086312Z 
2023-01-11T21:52:40.5086570Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5086682Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5086701Z 
2023-01-11T21:52:40.5086805Z OK (skipped=1)
2023-01-11T21:52:40.5086824Z 
2023-01-11T21:52:40.5086945Z Generating XML reports...
2023-01-11T21:52:40.5087389Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213248.xml
2023-01-11T21:52:40.5087834Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5088001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5088440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5088631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5088651Z 
2023-01-11T21:52:40.5088759Z Running tests...
2023-01-11T21:52:40.5089017Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5089320Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5089612Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:52:40.5089631Z 
2023-01-11T21:52:40.5089895Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5090008Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5090027Z 
2023-01-11T21:52:40.5090116Z OK (skipped=1)
2023-01-11T21:52:40.5090135Z 
2023-01-11T21:52:40.5090261Z Generating XML reports...
2023-01-11T21:52:40.5090712Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213251.xml
2023-01-11T21:52:40.5091140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5091320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5091704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5091894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5091914Z 
2023-01-11T21:52:40.5092022Z Running tests...
2023-01-11T21:52:40.5092288Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5092583Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5092892Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T21:52:40.5092915Z 
2023-01-11T21:52:40.5093175Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5093285Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5093304Z 
2023-01-11T21:52:40.5093410Z OK (skipped=1)
2023-01-11T21:52:40.5093429Z 
2023-01-11T21:52:40.5093550Z Generating XML reports...
2023-01-11T21:52:40.5093999Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213253.xml
2023-01-11T21:52:40.5094370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5094549Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5094913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5095109Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5095128Z 
2023-01-11T21:52:40.5095236Z Running tests...
2023-01-11T21:52:40.5095498Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5095809Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5096106Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T21:52:40.5096126Z 
2023-01-11T21:52:40.5096386Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5096497Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5096517Z 
2023-01-11T21:52:40.5096605Z OK (skipped=1)
2023-01-11T21:52:40.5096699Z 
2023-01-11T21:52:40.5096811Z Generating XML reports...
2023-01-11T21:52:40.5097267Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213255.xml
2023-01-11T21:52:40.5097702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5097878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5098258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5098451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5098470Z 
2023-01-11T21:52:40.5098578Z Running tests...
2023-01-11T21:52:40.5098837Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5099134Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5099440Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T21:52:40.5099463Z 
2023-01-11T21:52:40.5099724Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5099834Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5099854Z 
2023-01-11T21:52:40.5099961Z OK (skipped=1)
2023-01-11T21:52:40.5099980Z 
2023-01-11T21:52:40.5100101Z Generating XML reports...
2023-01-11T21:52:40.5100548Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213258.xml
2023-01-11T21:52:40.5100918Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5101092Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5101454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5101646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5101664Z 
2023-01-11T21:52:40.5101775Z Running tests...
2023-01-11T21:52:40.5102037Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5102350Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5102615Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5102837Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10172
2023-01-11T21:52:40.5103059Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10173
2023-01-11T21:52:40.5103413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5103591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5103976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5104168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5104535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5104712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5105086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5105278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5105522Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5105753Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5106211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5106622Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5106911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5107143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5107386Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.5107630Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.5108032Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5108434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5108520Z ok (5.768s)
2023-01-11T21:52:40.5108539Z 
2023-01-11T21:52:40.5108801Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5108918Z Ran 1 test in 5.768s
2023-01-11T21:52:40.5108938Z 
2023-01-11T21:52:40.5109032Z OK
2023-01-11T21:52:40.5109052Z 
2023-01-11T21:52:40.5109176Z Generating XML reports...
2023-01-11T21:52:40.5109628Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213300.xml
2023-01-11T21:52:40.5109997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5110175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5110541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5110741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5110761Z 
2023-01-11T21:52:40.5110871Z Running tests...
2023-01-11T21:52:40.5111135Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5111453Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5111717Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5111938Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10293
2023-01-11T21:52:40.5112157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10294
2023-01-11T21:52:40.5112531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5112689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5113069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5113259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5113626Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5113805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5114179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5114367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5114615Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5114842Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5115295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5115705Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5115982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5116210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5116359Z skip: Need at least 3 CUDA devices (4.231s)
2023-01-11T21:52:40.5116379Z 
2023-01-11T21:52:40.5116644Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5116756Z Ran 1 test in 4.231s
2023-01-11T21:52:40.5116776Z 
2023-01-11T21:52:40.5116883Z OK (skipped=1)
2023-01-11T21:52:40.5116901Z 
2023-01-11T21:52:40.5117007Z Generating XML reports...
2023-01-11T21:52:40.5117463Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213308.xml
2023-01-11T21:52:40.5117839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5118021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5118404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5118596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5118615Z 
2023-01-11T21:52:40.5118724Z Running tests...
2023-01-11T21:52:40.5118983Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5119297Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5119531Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2023-01-11T21:52:40.5119551Z 
2023-01-11T21:52:40.5119814Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5119928Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5119947Z 
2023-01-11T21:52:40.5120055Z OK (skipped=1)
2023-01-11T21:52:40.5120073Z 
2023-01-11T21:52:40.5120197Z Generating XML reports...
2023-01-11T21:52:40.5120648Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213315.xml
2023-01-11T21:52:40.5121017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5121192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5121570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5121746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5121766Z 
2023-01-11T21:52:40.5121873Z Running tests...
2023-01-11T21:52:40.5122133Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5122450Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5122695Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5122920Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10435
2023-01-11T21:52:40.5123139Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10436
2023-01-11T21:52:40.5123509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5123667Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5124049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5124555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5125023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5125205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5125590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5125839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5126087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5126335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5126727Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5127128Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5127364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5127595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5127701Z ok (4.918s)
2023-01-11T21:52:40.5127721Z 
2023-01-11T21:52:40.5127985Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5128097Z Ran 1 test in 4.919s
2023-01-11T21:52:40.5128117Z 
2023-01-11T21:52:40.5128211Z OK
2023-01-11T21:52:40.5128231Z 
2023-01-11T21:52:40.5128336Z Generating XML reports...
2023-01-11T21:52:40.5128794Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213318.xml
2023-01-11T21:52:40.5129165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5129343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5129728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5129921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5129943Z 
2023-01-11T21:52:40.5130052Z Running tests...
2023-01-11T21:52:40.5130313Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5130631Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5130866Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5131086Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10544
2023-01-11T21:52:40.5131306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10545
2023-01-11T21:52:40.5131679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5131857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5132237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5132432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5132843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5133003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5133381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5133568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5133816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5134065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5134521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5134932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5135211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5135441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5135525Z ok (5.837s)
2023-01-11T21:52:40.5135545Z 
2023-01-11T21:52:40.5135812Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5135923Z Ran 1 test in 5.837s
2023-01-11T21:52:40.5135942Z 
2023-01-11T21:52:40.5136036Z OK
2023-01-11T21:52:40.5136056Z 
2023-01-11T21:52:40.5136180Z Generating XML reports...
2023-01-11T21:52:40.5136636Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213325.xml
2023-01-11T21:52:40.5137008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5137189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5137575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5137751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5137770Z 
2023-01-11T21:52:40.5137878Z Running tests...
2023-01-11T21:52:40.5138143Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5138459Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5138722Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5138947Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10655
2023-01-11T21:52:40.5139167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10656
2023-01-11T21:52:40.5139544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5139702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5140082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5140274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5140640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5140814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5141191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5141380Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5141628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5141858Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5142262Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5142664Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5142895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5143125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5143461Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.5143709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.5144108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5144549Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5144651Z ok (4.996s)
2023-01-11T21:52:40.5144671Z 
2023-01-11T21:52:40.5144920Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5145034Z Ran 1 test in 4.997s
2023-01-11T21:52:40.5145053Z 
2023-01-11T21:52:40.5145145Z OK
2023-01-11T21:52:40.5145165Z 
2023-01-11T21:52:40.5145289Z Generating XML reports...
2023-01-11T21:52:40.5145748Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213333.xml
2023-01-11T21:52:40.5146128Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5146306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5146691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5146866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5146904Z 
2023-01-11T21:52:40.5146996Z Running tests...
2023-01-11T21:52:40.5147262Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5147578Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5147847Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5148070Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10770
2023-01-11T21:52:40.5148292Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10771
2023-01-11T21:52:40.5148667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5148845Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5149209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5149401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5149762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5149937Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5150312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5150508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5150758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5151160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5151392Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5151788Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5152017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5152245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5152405Z skip: Skipped due to small world size. (4.247s)
2023-01-11T21:52:40.5152424Z 
2023-01-11T21:52:40.5152739Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5152857Z Ran 1 test in 4.247s
2023-01-11T21:52:40.5152876Z 
2023-01-11T21:52:40.5152982Z OK (skipped=1)
2023-01-11T21:52:40.5153002Z 
2023-01-11T21:52:40.5153169Z Generating XML reports...
2023-01-11T21:52:40.5153612Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213341.xml
2023-01-11T21:52:40.5153981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5154159Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5154537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5154730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5154750Z 
2023-01-11T21:52:40.5154859Z Running tests...
2023-01-11T21:52:40.5155127Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5155445Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5155682Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5155906Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10879
2023-01-11T21:52:40.5156124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10880
2023-01-11T21:52:40.5156499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5156677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5157057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5157250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5157627Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5157799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5158163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5158354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5158602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5158849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5159251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5159647Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5159884Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5160117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5160280Z skip: Skipped due to small world size. (4.249s)
2023-01-11T21:52:40.5160300Z 
2023-01-11T21:52:40.5160547Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5160662Z Ran 1 test in 4.249s
2023-01-11T21:52:40.5160681Z 
2023-01-11T21:52:40.5160788Z OK (skipped=1)
2023-01-11T21:52:40.5160807Z 
2023-01-11T21:52:40.5160931Z Generating XML reports...
2023-01-11T21:52:40.5161381Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213348.xml
2023-01-11T21:52:40.5161751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5161931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5162361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5162544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5162618Z 
2023-01-11T21:52:40.5162712Z Running tests...
2023-01-11T21:52:40.5162977Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5163295Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5163560Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5163781Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 10988
2023-01-11T21:52:40.5163999Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 10989
2023-01-11T21:52:40.5164610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5164795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5165166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5165361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5165724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5165896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5166271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5166460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5166709Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5166958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5167345Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5167748Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5167979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5168209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5168368Z skip: Skipped due to small world size. (4.116s)
2023-01-11T21:52:40.5168387Z 
2023-01-11T21:52:40.5168655Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5168767Z Ran 1 test in 4.116s
2023-01-11T21:52:40.5168786Z 
2023-01-11T21:52:40.5168892Z OK (skipped=1)
2023-01-11T21:52:40.5168911Z 
2023-01-11T21:52:40.5169039Z Generating XML reports...
2023-01-11T21:52:40.5169477Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213354.xml
2023-01-11T21:52:40.5169857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5170034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5170415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5170607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5170627Z 
2023-01-11T21:52:40.5170736Z Running tests...
2023-01-11T21:52:40.5170998Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5171314Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5171660Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5171875Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11097
2023-01-11T21:52:40.5172163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11098
2023-01-11T21:52:40.5172538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5172714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5173091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5173283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5173647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5173819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5174189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5174383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5174634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5174880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5175282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5175676Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5175908Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5176143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5176384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.5176613Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.5177014Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5177408Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5177511Z ok (5.356s)
2023-01-11T21:52:40.5177531Z 
2023-01-11T21:52:40.5177793Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5177905Z Ran 1 test in 5.356s
2023-01-11T21:52:40.5177925Z 
2023-01-11T21:52:40.5178017Z OK
2023-01-11T21:52:40.5178038Z 
2023-01-11T21:52:40.5178161Z Generating XML reports...
2023-01-11T21:52:40.5178622Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213401.xml
2023-01-11T21:52:40.5178981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5179163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5179547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5179737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5179757Z 
2023-01-11T21:52:40.5179866Z Running tests...
2023-01-11T21:52:40.5180130Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5180445Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5180909Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... skip: Requires file:// initialization method. Both tcp:// and env:// rely on the TCP store for which reinitialization has proven racy. (0.002s)
2023-01-11T21:52:40.5180932Z 
2023-01-11T21:52:40.5181202Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5181341Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5181360Z 
2023-01-11T21:52:40.5181470Z OK (skipped=1)
2023-01-11T21:52:40.5181489Z 
2023-01-11T21:52:40.5181612Z Generating XML reports...
2023-01-11T21:52:40.5182066Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213409.xml
2023-01-11T21:52:40.5182440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5182617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5183000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5183197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5183218Z 
2023-01-11T21:52:40.5183326Z Running tests...
2023-01-11T21:52:40.5183567Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5183887Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5184152Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5184374Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11245
2023-01-11T21:52:40.5184594Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11246
2023-01-11T21:52:40.5184964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5185140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5185528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5185702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5186069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5186244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5186619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5186807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5187056Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5187304Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5187709Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5188109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5188326Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5188556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5188717Z skip: Skipped due to small world size. (4.247s)
2023-01-11T21:52:40.5188736Z 
2023-01-11T21:52:40.5189003Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5189115Z Ran 1 test in 4.247s
2023-01-11T21:52:40.5189134Z 
2023-01-11T21:52:40.5189242Z OK (skipped=1)
2023-01-11T21:52:40.5189260Z 
2023-01-11T21:52:40.5189383Z Generating XML reports...
2023-01-11T21:52:40.5189840Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213411.xml
2023-01-11T21:52:40.5190268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5190434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5190864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5191055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5191076Z 
2023-01-11T21:52:40.5191183Z Running tests...
2023-01-11T21:52:40.5191446Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5191762Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5192028Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5192249Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11354
2023-01-11T21:52:40.5192454Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11355
2023-01-11T21:52:40.5192826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5193005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5193387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5193577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5193945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5194119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5194497Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5194688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5194919Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5195166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5195569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5195968Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5196200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5196427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5196529Z ok (4.133s)
2023-01-11T21:52:40.5196549Z 
2023-01-11T21:52:40.5196814Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5196912Z Ran 1 test in 4.133s
2023-01-11T21:52:40.5196949Z 
2023-01-11T21:52:40.5197024Z OK
2023-01-11T21:52:40.5197043Z 
2023-01-11T21:52:40.5197165Z Generating XML reports...
2023-01-11T21:52:40.5197625Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213418.xml
2023-01-11T21:52:40.5198000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5198178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5198557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5198749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5198768Z 
2023-01-11T21:52:40.5198877Z Running tests...
2023-01-11T21:52:40.5199123Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5199487Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5199767Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5200030Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11463
2023-01-11T21:52:40.5200251Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11464
2023-01-11T21:52:40.5200628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5200805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5201187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5201360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5201731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5201905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5202289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5202480Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5202726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5202974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5203377Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5203775Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5203993Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5204446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5204564Z ok (4.240s)
2023-01-11T21:52:40.5204584Z 
2023-01-11T21:52:40.5204853Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5204967Z Ran 1 test in 4.240s
2023-01-11T21:52:40.5204986Z 
2023-01-11T21:52:40.5205079Z OK
2023-01-11T21:52:40.5205098Z 
2023-01-11T21:52:40.5205224Z Generating XML reports...
2023-01-11T21:52:40.5205680Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213425.xml
2023-01-11T21:52:40.5206054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5206215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5206600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5206794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5206816Z 
2023-01-11T21:52:40.5206924Z Running tests...
2023-01-11T21:52:40.5207190Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5207508Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5207785Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:52:40.5207805Z 
2023-01-11T21:52:40.5208067Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5208161Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5208198Z 
2023-01-11T21:52:40.5208288Z OK (skipped=1)
2023-01-11T21:52:40.5208307Z 
2023-01-11T21:52:40.5208430Z Generating XML reports...
2023-01-11T21:52:40.5208958Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213432.xml
2023-01-11T21:52:40.5209348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5209582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5209972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5210164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5210184Z 
2023-01-11T21:52:40.5210293Z Running tests...
2023-01-11T21:52:40.5210539Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5210854Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5211118Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2023-01-11T21:52:40.5211138Z 
2023-01-11T21:52:40.5211402Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5211515Z Ran 1 test in 0.003s
2023-01-11T21:52:40.5211537Z 
2023-01-11T21:52:40.5211645Z OK (skipped=1)
2023-01-11T21:52:40.5211664Z 
2023-01-11T21:52:40.5211789Z Generating XML reports...
2023-01-11T21:52:40.5212242Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213434.xml
2023-01-11T21:52:40.5212615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5212775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5213158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5213350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5213373Z 
2023-01-11T21:52:40.5213481Z Running tests...
2023-01-11T21:52:40.5213743Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5214062Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5214333Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:52:40.5214353Z 
2023-01-11T21:52:40.5214612Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5214703Z Ran 1 test in 0.003s
2023-01-11T21:52:40.5214741Z 
2023-01-11T21:52:40.5214830Z OK (skipped=1)
2023-01-11T21:52:40.5214848Z 
2023-01-11T21:52:40.5214971Z Generating XML reports...
2023-01-11T21:52:40.5215422Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213436.xml
2023-01-11T21:52:40.5215798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5215977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5216359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5216557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5216577Z 
2023-01-11T21:52:40.5216685Z Running tests...
2023-01-11T21:52:40.5216929Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5217243Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5217502Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:52:40.5217522Z 
2023-01-11T21:52:40.5217784Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5217898Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5217967Z 
2023-01-11T21:52:40.5218081Z OK (skipped=1)
2023-01-11T21:52:40.5218100Z 
2023-01-11T21:52:40.5218225Z Generating XML reports...
2023-01-11T21:52:40.5218685Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213439.xml
2023-01-11T21:52:40.5219112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5219270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5219653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5219847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5219867Z 
2023-01-11T21:52:40.5219975Z Running tests...
2023-01-11T21:52:40.5220236Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5220555Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5220823Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:52:40.5220846Z 
2023-01-11T21:52:40.5221109Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5221221Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5221241Z 
2023-01-11T21:52:40.5221330Z OK (skipped=1)
2023-01-11T21:52:40.5221349Z 
2023-01-11T21:52:40.5221475Z Generating XML reports...
2023-01-11T21:52:40.5221928Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213441.xml
2023-01-11T21:52:40.5222302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5222480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5222864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5223057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5223079Z 
2023-01-11T21:52:40.5223188Z Running tests...
2023-01-11T21:52:40.5223431Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5223744Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5224021Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:52:40.5224041Z 
2023-01-11T21:52:40.5224298Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5224411Z Ran 1 test in 0.003s
2023-01-11T21:52:40.5224430Z 
2023-01-11T21:52:40.5224536Z OK (skipped=1)
2023-01-11T21:52:40.5224555Z 
2023-01-11T21:52:40.5224678Z Generating XML reports...
2023-01-11T21:52:40.5225132Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213443.xml
2023-01-11T21:52:40.5225507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5225669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5226052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5226244Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5226263Z 
2023-01-11T21:52:40.5226371Z Running tests...
2023-01-11T21:52:40.5226632Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5226945Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5227212Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:52:40.5227281Z 
2023-01-11T21:52:40.5227556Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5227668Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5227729Z 
2023-01-11T21:52:40.5227820Z OK (skipped=1)
2023-01-11T21:52:40.5227838Z 
2023-01-11T21:52:40.5227961Z Generating XML reports...
2023-01-11T21:52:40.5228415Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213446.xml
2023-01-11T21:52:40.5228789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5228967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5229345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5229537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5229557Z 
2023-01-11T21:52:40.5229669Z Running tests...
2023-01-11T21:52:40.5229914Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5230233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5230500Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T21:52:40.5230520Z 
2023-01-11T21:52:40.5230778Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5230888Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5230907Z 
2023-01-11T21:52:40.5231015Z OK (skipped=1)
2023-01-11T21:52:40.5231034Z 
2023-01-11T21:52:40.5231157Z Generating XML reports...
2023-01-11T21:52:40.5231603Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213448.xml
2023-01-11T21:52:40.5231979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5232137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5232517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5232757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5232777Z 
2023-01-11T21:52:40.5232887Z Running tests...
2023-01-11T21:52:40.5233151Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5233465Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5233714Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5233936Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11836
2023-01-11T21:52:40.5234156Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11837
2023-01-11T21:52:40.5234512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5234689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5235075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5235267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5235637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5235813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5236187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5236375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5236654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5237069Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5237367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5237772Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5238005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5238235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5238573Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5238901Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5239242Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5239557Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5239910Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5240262Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5240595Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5240918Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5241249Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5241813Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5241834Z 
2023-01-11T21:52:40.5242182Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5242509Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5242827Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5243146Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5243472Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5243814Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5244161Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5244729Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5245056Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5245395Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5245740Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5246070Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5246396Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5246723Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5247120Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5247468Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5247857Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5248206Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5248550Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5248875Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5249202Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5249515Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5249861Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5250196Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5250546Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5250875Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5251200Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5251532Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5251862Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5252203Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5252535Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5252861Z STAGE:2023-01-11 21:34:54 11836:11836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5253187Z STAGE:2023-01-11 21:34:54 11837:11837 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5253519Z STAGE:2023-01-11 21:34:55 11837:11837 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5253846Z STAGE:2023-01-11 21:34:55 11836:11836 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5254191Z STAGE:2023-01-11 21:34:55 11837:11837 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5254531Z STAGE:2023-01-11 21:34:55 11836:11836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5254858Z STAGE:2023-01-11 21:34:55 11837:11837 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5255185Z STAGE:2023-01-11 21:34:55 11836:11836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5255502Z STAGE:2023-01-11 21:34:55 11836:11836 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5255835Z STAGE:2023-01-11 21:34:55 11837:11837 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5256180Z STAGE:2023-01-11 21:34:55 11836:11836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5256525Z STAGE:2023-01-11 21:34:55 11837:11837 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5256853Z STAGE:2023-01-11 21:34:55 11836:11836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5257176Z STAGE:2023-01-11 21:34:55 11837:11837 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5257559Z STAGE:2023-01-11 21:34:55 11837:11837 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5258129Z STAGE:2023-01-11 21:34:55 11836:11836 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:34:55 11837:11837 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5258192Z 
2023-01-11T21:52:40.5258543Z STAGE:2023-01-11 21:34:55 11836:11836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5258867Z STAGE:2023-01-11 21:34:55 11837:11837 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5259178Z STAGE:2023-01-11 21:34:55 11836:11836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5259512Z STAGE:2023-01-11 21:34:55 11836:11836 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5259838Z STAGE:2023-01-11 21:34:55 11837:11837 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5260186Z STAGE:2023-01-11 21:34:55 11836:11836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5260531Z STAGE:2023-01-11 21:34:55 11837:11837 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5260860Z STAGE:2023-01-11 21:34:55 11836:11836 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5261182Z STAGE:2023-01-11 21:34:55 11837:11837 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5261514Z STAGE:2023-01-11 21:34:55 11837:11837 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5261856Z STAGE:2023-01-11 21:34:55 11837:11837 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5262171Z STAGE:2023-01-11 21:34:55 11836:11836 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5262514Z STAGE:2023-01-11 21:34:55 11836:11836 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5262621Z ok (4.239s)
2023-01-11T21:52:40.5262641Z 
2023-01-11T21:52:40.5262910Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5263020Z Ran 1 test in 4.239s
2023-01-11T21:52:40.5263043Z 
2023-01-11T21:52:40.5263138Z OK
2023-01-11T21:52:40.5263157Z 
2023-01-11T21:52:40.5263282Z Generating XML reports...
2023-01-11T21:52:40.5263741Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213451.xml
2023-01-11T21:52:40.5264097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5264277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5264665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5264859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5264878Z 
2023-01-11T21:52:40.5264990Z Running tests...
2023-01-11T21:52:40.5265253Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5265567Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5265827Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5266588Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.637s)
2023-01-11T21:52:40.5266610Z 
2023-01-11T21:52:40.5266875Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5266970Z Ran 1 test in 1.637s
2023-01-11T21:52:40.5266990Z 
2023-01-11T21:52:40.5267098Z OK (skipped=1)
2023-01-11T21:52:40.5267116Z 
2023-01-11T21:52:40.5267293Z Generating XML reports...
2023-01-11T21:52:40.5267761Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213457.xml
2023-01-11T21:52:40.5268187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5268364Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5268747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5268939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5268959Z 
2023-01-11T21:52:40.5269049Z Running tests...
2023-01-11T21:52:40.5269314Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5269629Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5269901Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5270123Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 11983
2023-01-11T21:52:40.5270350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 11984
2023-01-11T21:52:40.5270726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5270902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5271285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5271460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5271830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5272004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5272384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5272578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5272827Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5273075Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5273479Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5273862Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5274097Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5274330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5274571Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.5274817Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.5275216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5275610Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5275946Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5276273Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5276592Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5276976Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5277336Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5277730Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5278058Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5278380Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5278715Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5279043Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5279389Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5279721Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5280049Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5280378Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5280713Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5281058Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5281391Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5281737Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5282067Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5282390Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5282707Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5283270Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5283290Z 
2023-01-11T21:52:40.5283637Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5283964Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5284504Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5284855Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5285405Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5285429Z 
2023-01-11T21:52:40.5285775Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5286101Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5286423Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5286755Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5287066Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5287483Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5287843Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5288270Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5288590Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5288923Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5289268Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5289600Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5289943Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5290254Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5290580Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5290917Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5291249Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5291593Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5291937Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5292263Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5292584Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5292919Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5293226Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5293576Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5293920Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5294248Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5294570Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5294903Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5295232Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5295572Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5295918Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5296228Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5296546Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5296882Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5297226Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5297554Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5297946Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5298282Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5298655Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5298988Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5299312Z STAGE:2023-01-11 21:35:05 11984:11984 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5299642Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5299984Z STAGE:2023-01-11 21:35:05 11983:11983 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5300087Z ok (4.327s)
2023-01-11T21:52:40.5300107Z 
2023-01-11T21:52:40.5300376Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5300490Z Ran 1 test in 4.327s
2023-01-11T21:52:40.5300510Z 
2023-01-11T21:52:40.5300602Z OK
2023-01-11T21:52:40.5300621Z 
2023-01-11T21:52:40.5300748Z Generating XML reports...
2023-01-11T21:52:40.5301189Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213502.xml
2023-01-11T21:52:40.5301602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5301782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5302166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5302379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5302399Z 
2023-01-11T21:52:40.5302509Z Running tests...
2023-01-11T21:52:40.5302784Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5303104Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5303363Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5303570Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12102
2023-01-11T21:52:40.5303793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12103
2023-01-11T21:52:40.5304168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5304343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5304727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5304920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5305293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5305469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5305832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5306025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5306270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5306517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5306921Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5307320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5307604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5307844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5308063Z skip: Skipped due to small world size. (4.125s)
2023-01-11T21:52:40.5308083Z 
2023-01-11T21:52:40.5308335Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5308447Z Ran 1 test in 4.125s
2023-01-11T21:52:40.5308466Z 
2023-01-11T21:52:40.5308572Z OK (skipped=1)
2023-01-11T21:52:40.5308591Z 
2023-01-11T21:52:40.5308716Z Generating XML reports...
2023-01-11T21:52:40.5309171Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213508.xml
2023-01-11T21:52:40.5309545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5309723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5310111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5310305Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5310327Z 
2023-01-11T21:52:40.5310418Z Running tests...
2023-01-11T21:52:40.5310679Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5310995Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5311261Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5311484Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12211
2023-01-11T21:52:40.5311705Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12212
2023-01-11T21:52:40.5312082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5312259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5312623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5312823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5313191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5313368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5313743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5313933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5314183Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5314431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5314833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5315217Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5315448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5315678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5316459Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1478: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T21:52:40.5316574Z   warnings.warn(
2023-01-11T21:52:40.5317394Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1478: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T21:52:40.5317548Z   warnings.warn(
2023-01-11T21:52:40.5317649Z ok (5.123s)
2023-01-11T21:52:40.5317668Z 
2023-01-11T21:52:40.5317936Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5318030Z Ran 1 test in 5.123s
2023-01-11T21:52:40.5318068Z 
2023-01-11T21:52:40.5318143Z OK
2023-01-11T21:52:40.5318162Z 
2023-01-11T21:52:40.5318287Z Generating XML reports...
2023-01-11T21:52:40.5318742Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213515.xml
2023-01-11T21:52:40.5319116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5319297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5319676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5319875Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5319894Z 
2023-01-11T21:52:40.5320003Z Running tests...
2023-01-11T21:52:40.5320247Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5320564Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5320832Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5321587Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.603s)
2023-01-11T21:52:40.5321607Z 
2023-01-11T21:52:40.5321871Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5321987Z Ran 1 test in 1.603s
2023-01-11T21:52:40.5322005Z 
2023-01-11T21:52:40.5322113Z OK (skipped=1)
2023-01-11T21:52:40.5322132Z 
2023-01-11T21:52:40.5322257Z Generating XML reports...
2023-01-11T21:52:40.5322713Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213523.xml
2023-01-11T21:52:40.5323087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5323246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5323625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5323821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5323841Z 
2023-01-11T21:52:40.5323950Z Running tests...
2023-01-11T21:52:40.5324387Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5324729Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5325046Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5325803Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.600s)
2023-01-11T21:52:40.5325823Z 
2023-01-11T21:52:40.5326084Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5326177Z Ran 1 test in 1.600s
2023-01-11T21:52:40.5326288Z 
2023-01-11T21:52:40.5326384Z OK (skipped=1)
2023-01-11T21:52:40.5326403Z 
2023-01-11T21:52:40.5326528Z Generating XML reports...
2023-01-11T21:52:40.5326985Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213527.xml
2023-01-11T21:52:40.5327421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5327601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5327980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5328174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5328194Z 
2023-01-11T21:52:40.5328303Z Running tests...
2023-01-11T21:52:40.5328544Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5328861Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5329185Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5329935Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.615s)
2023-01-11T21:52:40.5329955Z 
2023-01-11T21:52:40.5330215Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5330328Z Ran 1 test in 1.615s
2023-01-11T21:52:40.5330347Z 
2023-01-11T21:52:40.5330453Z OK (skipped=1)
2023-01-11T21:52:40.5330472Z 
2023-01-11T21:52:40.5330596Z Generating XML reports...
2023-01-11T21:52:40.5331054Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213531.xml
2023-01-11T21:52:40.5331431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5331594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5331976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5332167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5332186Z 
2023-01-11T21:52:40.5332295Z Running tests...
2023-01-11T21:52:40.5332558Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5332915Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5333196Z   test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5333422Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12424
2023-01-11T21:52:40.5333625Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12425
2023-01-11T21:52:40.5334004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5334176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5334560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5334760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5335124Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5335296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5335720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5335919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5336165Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5336435Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5336840Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5337237Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5337469Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5337697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5338484Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T21:52:40.5338599Z   warnings.warn(
2023-01-11T21:52:40.5339380Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T21:52:40.5339491Z   warnings.warn(
2023-01-11T21:52:40.5339727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5339942Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5340178Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5340403Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5340637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5340871Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5341095Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5341321Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5341420Z ok (7.144s)
2023-01-11T21:52:40.5341440Z 
2023-01-11T21:52:40.5341707Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5341803Z Ran 1 test in 7.144s
2023-01-11T21:52:40.5341823Z 
2023-01-11T21:52:40.5341915Z OK
2023-01-11T21:52:40.5341935Z 
2023-01-11T21:52:40.5342059Z Generating XML reports...
2023-01-11T21:52:40.5342518Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213535.xml
2023-01-11T21:52:40.5342893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5343073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5343454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5343646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5343666Z 
2023-01-11T21:52:40.5343756Z Running tests...
2023-01-11T21:52:40.5344018Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5344331Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5344703Z   test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5344932Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12539
2023-01-11T21:52:40.5345148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12540
2023-01-11T21:52:40.5345569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5345742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5346120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5346293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5346655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5346828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5347205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5347396Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5347643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5347889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5348288Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5348684Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5348898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5349126Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5349910Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T21:52:40.5350025Z   warnings.warn(
2023-01-11T21:52:40.5350805Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T21:52:40.5350914Z   warnings.warn(
2023-01-11T21:52:40.5351151Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5351383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5351616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5351832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5351935Z ok (6.130s)
2023-01-11T21:52:40.5351956Z 
2023-01-11T21:52:40.5352222Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5352332Z Ran 1 test in 6.130s
2023-01-11T21:52:40.5352352Z 
2023-01-11T21:52:40.5352444Z OK
2023-01-11T21:52:40.5352464Z 
2023-01-11T21:52:40.5352586Z Generating XML reports...
2023-01-11T21:52:40.5353038Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213545.xml
2023-01-11T21:52:40.5353411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5353586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5354001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5354198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5354257Z 
2023-01-11T21:52:40.5354367Z Running tests...
2023-01-11T21:52:40.5354633Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5354944Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5355240Z   test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5355456Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12654
2023-01-11T21:52:40.5355674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12655
2023-01-11T21:52:40.5356031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5356205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5356584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5356778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5357147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5357321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5357694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5357883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5358127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5358357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5358761Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5359164Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5359394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5359623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5360404Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T21:52:40.5360519Z   warnings.warn(
2023-01-11T21:52:40.5361296Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T21:52:40.5361408Z   warnings.warn(
2023-01-11T21:52:40.5361506Z ok (6.101s)
2023-01-11T21:52:40.5361526Z 
2023-01-11T21:52:40.5361774Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5361884Z Ran 1 test in 6.101s
2023-01-11T21:52:40.5361903Z 
2023-01-11T21:52:40.5361994Z OK
2023-01-11T21:52:40.5362014Z 
2023-01-11T21:52:40.5362136Z Generating XML reports...
2023-01-11T21:52:40.5362590Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213553.xml
2023-01-11T21:52:40.5363009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5363192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5363577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5363800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5363837Z 
2023-01-11T21:52:40.5363926Z Running tests...
2023-01-11T21:52:40.5364404Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5364745Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5365014Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5365235Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12771
2023-01-11T21:52:40.5365457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12772
2023-01-11T21:52:40.5365831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5366009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5366373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5366562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5366930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5367102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5367477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5367664Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5367914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5368162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5368568Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5368952Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5369182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5369410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5369646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5369880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5369985Z ok (5.539s)
2023-01-11T21:52:40.5370006Z 
2023-01-11T21:52:40.5370273Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5370386Z Ran 1 test in 5.540s
2023-01-11T21:52:40.5370406Z 
2023-01-11T21:52:40.5370482Z OK
2023-01-11T21:52:40.5370500Z 
2023-01-11T21:52:40.5370622Z Generating XML reports...
2023-01-11T21:52:40.5371078Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213602.xml
2023-01-11T21:52:40.5371450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5371624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5372006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5372197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5372293Z 
2023-01-11T21:52:40.5372408Z Running tests...
2023-01-11T21:52:40.5372674Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5373063Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5373340Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5373559Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 12886
2023-01-11T21:52:40.5373778Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 12887
2023-01-11T21:52:40.5374148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5374321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5374704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5374894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5375243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5375422Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5375794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5375981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5376227Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5376471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5376873Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5377274Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5377503Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5377718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5377953Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5378181Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5378405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5378640Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5378740Z ok (5.530s)
2023-01-11T21:52:40.5378760Z 
2023-01-11T21:52:40.5379031Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5379144Z Ran 1 test in 5.530s
2023-01-11T21:52:40.5379164Z 
2023-01-11T21:52:40.5379238Z OK
2023-01-11T21:52:40.5379272Z 
2023-01-11T21:52:40.5379379Z Generating XML reports...
2023-01-11T21:52:40.5379840Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213610.xml
2023-01-11T21:52:40.5380213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5380388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5380764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5380956Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5380975Z 
2023-01-11T21:52:40.5381081Z Running tests...
2023-01-11T21:52:40.5381341Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5381692Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5381974Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5382772Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.588s)
2023-01-11T21:52:40.5382792Z 
2023-01-11T21:52:40.5383052Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5383163Z Ran 1 test in 1.588s
2023-01-11T21:52:40.5383183Z 
2023-01-11T21:52:40.5383290Z OK (skipped=1)
2023-01-11T21:52:40.5383308Z 
2023-01-11T21:52:40.5383431Z Generating XML reports...
2023-01-11T21:52:40.5383884Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213618.xml
2023-01-11T21:52:40.5384255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5384434Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5384800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5384990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5385010Z 
2023-01-11T21:52:40.5385119Z Running tests...
2023-01-11T21:52:40.5385379Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5385692Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5385981Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5386730Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.615s)
2023-01-11T21:52:40.5386754Z 
2023-01-11T21:52:40.5387011Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5387123Z Ran 1 test in 1.615s
2023-01-11T21:52:40.5387142Z 
2023-01-11T21:52:40.5387230Z OK (skipped=1)
2023-01-11T21:52:40.5387265Z 
2023-01-11T21:52:40.5387371Z Generating XML reports...
2023-01-11T21:52:40.5387822Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213622.xml
2023-01-11T21:52:40.5388193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5388371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5388751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5388944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5388963Z 
2023-01-11T21:52:40.5389069Z Running tests...
2023-01-11T21:52:40.5389327Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5389627Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5389911Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5390131Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13069
2023-01-11T21:52:40.5390348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13070
2023-01-11T21:52:40.5390765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5390946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5391388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5391629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5391986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5392160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5392533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5392724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5392969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5393220Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5393622Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5394023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5394252Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5394462Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5394672Z <class 'torch.nn.parameter.Parameter'>
2023-01-11T21:52:40.5394772Z ok (5.026s)
2023-01-11T21:52:40.5394792Z 
2023-01-11T21:52:40.5395053Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5395164Z Ran 1 test in 5.026s
2023-01-11T21:52:40.5395183Z 
2023-01-11T21:52:40.5395275Z OK
2023-01-11T21:52:40.5395298Z 
2023-01-11T21:52:40.5395420Z Generating XML reports...
2023-01-11T21:52:40.5395872Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213627.xml
2023-01-11T21:52:40.5396231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5396408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5396786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5396977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5396996Z 
2023-01-11T21:52:40.5397102Z Running tests...
2023-01-11T21:52:40.5397360Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5397673Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5397981Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5398202Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13180
2023-01-11T21:52:40.5398406Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13181
2023-01-11T21:52:40.5398777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5398951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5399329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5399518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5399882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5400109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5400496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5400720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5400966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5401210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5401647Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5402049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5402278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5402510Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5402611Z ok (5.155s)
2023-01-11T21:52:40.5402631Z 
2023-01-11T21:52:40.5402895Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5402988Z Ran 1 test in 5.156s
2023-01-11T21:52:40.5403008Z 
2023-01-11T21:52:40.5403098Z OK
2023-01-11T21:52:40.5403118Z 
2023-01-11T21:52:40.5403241Z Generating XML reports...
2023-01-11T21:52:40.5403692Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213634.xml
2023-01-11T21:52:40.5404060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5404414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5404813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5405008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5405028Z 
2023-01-11T21:52:40.5405136Z Running tests...
2023-01-11T21:52:40.5405386Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5405700Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5405964Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5406182Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13291
2023-01-11T21:52:40.5406398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13292
2023-01-11T21:52:40.5406769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5406944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5407332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5407508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5407879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5408052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5408432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5408622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5408868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5409113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5409587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5409998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5410271Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5410499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5410734Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5410968Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5411070Z ok (5.625s)
2023-01-11T21:52:40.5411090Z 
2023-01-11T21:52:40.5411358Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5411469Z Ran 1 test in 5.626s
2023-01-11T21:52:40.5411488Z 
2023-01-11T21:52:40.5411581Z OK
2023-01-11T21:52:40.5411601Z 
2023-01-11T21:52:40.5411710Z Generating XML reports...
2023-01-11T21:52:40.5412165Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213642.xml
2023-01-11T21:52:40.5412540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5412716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5413099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5413290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5413309Z 
2023-01-11T21:52:40.5413415Z Running tests...
2023-01-11T21:52:40.5413675Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5413989Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5414266Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5414484Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13406
2023-01-11T21:52:40.5414702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13407
2023-01-11T21:52:40.5415075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5415249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5415632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5415822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5416190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5416351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5416734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5416927Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5417171Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5417416Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5417818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5418216Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5418446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5418724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5419524Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:52:40.5419669Z ok (5.502s)
2023-01-11T21:52:40.5419690Z 
2023-01-11T21:52:40.5419939Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5420050Z Ran 1 test in 5.503s
2023-01-11T21:52:40.5420069Z 
2023-01-11T21:52:40.5420160Z OK
2023-01-11T21:52:40.5420179Z 
2023-01-11T21:52:40.5420302Z Generating XML reports...
2023-01-11T21:52:40.5420758Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213650.xml
2023-01-11T21:52:40.5421131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5421310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5421691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5421882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5421902Z 
2023-01-11T21:52:40.5421992Z Running tests...
2023-01-11T21:52:40.5422251Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5422564Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5422851Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5423608Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.627s)
2023-01-11T21:52:40.5423631Z 
2023-01-11T21:52:40.5423888Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5423998Z Ran 1 test in 1.627s
2023-01-11T21:52:40.5424017Z 
2023-01-11T21:52:40.5424121Z OK (skipped=1)
2023-01-11T21:52:40.5424140Z 
2023-01-11T21:52:40.5424261Z Generating XML reports...
2023-01-11T21:52:40.5424694Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213658.xml
2023-01-11T21:52:40.5425066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5425245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5425624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5425816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5425835Z 
2023-01-11T21:52:40.5425941Z Running tests...
2023-01-11T21:52:40.5426202Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5426518Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5426778Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5426983Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13555
2023-01-11T21:52:40.5427198Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13556
2023-01-11T21:52:40.5427614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5427795Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5428218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5428407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5428772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5428945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5429305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5429491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5429741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5429984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5430389Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5430787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5431014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5431236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5432140Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:52:40.5433089Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:52:40.5434278Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1134.)
2023-01-11T21:52:40.5434510Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2023-01-11T21:52:40.5435680Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1134.)
2023-01-11T21:52:40.5435916Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2023-01-11T21:52:40.5436154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5436457Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5437370Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:52:40.5438324Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:52:40.5439215Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:52:40.5440099Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:52:40.5440977Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:52:40.5441859Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:52:40.5442740Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:52:40.5443632Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:52:40.5444738Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:52:40.5445634Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T21:52:40.5445721Z ok (4.142s)
2023-01-11T21:52:40.5445757Z 
2023-01-11T21:52:40.5446079Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5446202Z Ran 1 test in 4.143s
2023-01-11T21:52:40.5446222Z 
2023-01-11T21:52:40.5446314Z OK
2023-01-11T21:52:40.5446381Z 
2023-01-11T21:52:40.5446506Z Generating XML reports...
2023-01-11T21:52:40.5446969Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213702.xml
2023-01-11T21:52:40.5447340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5447517Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5447903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5448080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5448100Z 
2023-01-11T21:52:40.5448207Z Running tests...
2023-01-11T21:52:40.5448479Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5448797Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5449050Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5449797Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.621s)
2023-01-11T21:52:40.5449817Z 
2023-01-11T21:52:40.5450078Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5450190Z Ran 1 test in 1.621s
2023-01-11T21:52:40.5450210Z 
2023-01-11T21:52:40.5450317Z OK (skipped=1)
2023-01-11T21:52:40.5450335Z 
2023-01-11T21:52:40.5450440Z Generating XML reports...
2023-01-11T21:52:40.5450898Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213709.xml
2023-01-11T21:52:40.5451270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5451448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5451830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5452020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5452040Z 
2023-01-11T21:52:40.5452226Z Running tests...
2023-01-11T21:52:40.5452527Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5452884Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5453144Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5453402Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13702
2023-01-11T21:52:40.5453774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13703
2023-01-11T21:52:40.5454214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5454428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5454846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5455076Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5455432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5455641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5456103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5456340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5456760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5457043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5457490Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5457927Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5458193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5458410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5459246Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1331: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2023-01-11T21:52:40.5459632Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2023-01-11T21:52:40.5460463Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1331: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2023-01-11T21:52:40.5460867Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2023-01-11T21:52:40.5461007Z ok (5.612s)
2023-01-11T21:52:40.5461027Z 
2023-01-11T21:52:40.5461330Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5461480Z Ran 1 test in 5.612s
2023-01-11T21:52:40.5461500Z 
2023-01-11T21:52:40.5461631Z OK
2023-01-11T21:52:40.5461652Z 
2023-01-11T21:52:40.5461820Z Generating XML reports...
2023-01-11T21:52:40.5462264Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213713.xml
2023-01-11T21:52:40.5462677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5462889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5463382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5463616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5463636Z 
2023-01-11T21:52:40.5463780Z Running tests...
2023-01-11T21:52:40.5464086Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5464453Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5464766Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5465505Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.619s)
2023-01-11T21:52:40.5465580Z 
2023-01-11T21:52:40.5465828Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5465975Z Ran 1 test in 1.619s
2023-01-11T21:52:40.5465994Z 
2023-01-11T21:52:40.5466172Z OK (skipped=1)
2023-01-11T21:52:40.5466191Z 
2023-01-11T21:52:40.5466351Z Generating XML reports...
2023-01-11T21:52:40.5466887Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213721.xml
2023-01-11T21:52:40.5467353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5467566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5467988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5468165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5468271Z 
2023-01-11T21:52:40.5468364Z Running tests...
2023-01-11T21:52:40.5468662Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5469054Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5469376Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5470161Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.615s)
2023-01-11T21:52:40.5470185Z 
2023-01-11T21:52:40.5470482Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5470628Z Ran 1 test in 1.615s
2023-01-11T21:52:40.5470647Z 
2023-01-11T21:52:40.5470785Z OK (skipped=1)
2023-01-11T21:52:40.5470804Z 
2023-01-11T21:52:40.5470963Z Generating XML reports...
2023-01-11T21:52:40.5471404Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213725.xml
2023-01-11T21:52:40.5471815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5472073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5472495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5472726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5472746Z 
2023-01-11T21:52:40.5472887Z Running tests...
2023-01-11T21:52:40.5473191Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5473548Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5473907Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5474115Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 13915
2023-01-11T21:52:40.5474377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 13916
2023-01-11T21:52:40.5474829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5475046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5475474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5475701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5476104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5476313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5476673Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5476910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5477242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5477562Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5478054Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5478486Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5478754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5479060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5479342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.5479570Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.5480015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5480447Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5480757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5481029Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5481292Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5481562Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5481699Z ok (5.848s)
2023-01-11T21:52:40.5481719Z 
2023-01-11T21:52:40.5482022Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5482120Z Ran 1 test in 5.848s
2023-01-11T21:52:40.5482140Z 
2023-01-11T21:52:40.5482271Z OK
2023-01-11T21:52:40.5482291Z 
2023-01-11T21:52:40.5482447Z Generating XML reports...
2023-01-11T21:52:40.5482973Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213729.xml
2023-01-11T21:52:40.5483397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5483623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5484045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5484502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5484524Z 
2023-01-11T21:52:40.5484625Z Running tests...
2023-01-11T21:52:40.5484934Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5485290Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5485604Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5485906Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14036
2023-01-11T21:52:40.5486173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14037
2023-01-11T21:52:40.5486590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5486804Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5487227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5487406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5487809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5488098Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5488533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5488863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5489154Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5489471Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5489921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5490306Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5490575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5490843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5491167Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T21:52:40.5491484Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T21:52:40.5491792Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5492063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5492333Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5492600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5492862Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T21:52:40.5493177Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T21:52:40.5493497Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T21:52:40.5493803Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T21:52:40.5494075Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5494374Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5494645Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5494950Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5495262Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T21:52:40.5495524Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T21:52:40.5495846Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2023-01-11T21:52:40.5496150Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2023-01-11T21:52:40.5496417Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5496683Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5496978Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5497247Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5497384Z ok (6.346s)
2023-01-11T21:52:40.5497404Z 
2023-01-11T21:52:40.5497775Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5497879Z Ran 1 test in 6.346s
2023-01-11T21:52:40.5497898Z 
2023-01-11T21:52:40.5498026Z OK
2023-01-11T21:52:40.5498082Z 
2023-01-11T21:52:40.5498242Z Generating XML reports...
2023-01-11T21:52:40.5498742Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213738.xml
2023-01-11T21:52:40.5499153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5499442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5499878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5500105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5500127Z 
2023-01-11T21:52:40.5500218Z Running tests...
2023-01-11T21:52:40.5500528Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5500877Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5501186Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5501995Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.606s)
2023-01-11T21:52:40.5502017Z 
2023-01-11T21:52:40.5502317Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5502511Z Ran 1 test in 1.606s
2023-01-11T21:52:40.5502532Z 
2023-01-11T21:52:40.5502675Z OK (skipped=1)
2023-01-11T21:52:40.5502694Z 
2023-01-11T21:52:40.5502852Z Generating XML reports...
2023-01-11T21:52:40.5503344Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213747.xml
2023-01-11T21:52:40.5503708Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5503923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5515534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5515766Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5515788Z 
2023-01-11T21:52:40.5515900Z Running tests...
2023-01-11T21:52:40.5516206Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5516532Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5516818Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5517047Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14185
2023-01-11T21:52:40.5517250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14186
2023-01-11T21:52:40.5517640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5517820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5518207Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5518405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5518782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5518962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5519443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5519649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5519935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5520185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5520601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5521005Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5521240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5521476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5522037Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:52:40.5522592Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T21:52:40.5522834Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5523071Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5523350Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2023-01-11T21:52:40.5523609Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2023-01-11T21:52:40.5523913Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2023-01-11T21:52:40.5524445Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2023-01-11T21:52:40.5524798Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2023-01-11T21:52:40.5525134Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2023-01-11T21:52:40.5525467Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2023-01-11T21:52:40.5525794Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2023-01-11T21:52:40.5526035Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5526270Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5526355Z ok (5.641s)
2023-01-11T21:52:40.5526394Z 
2023-01-11T21:52:40.5526657Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5526774Z Ran 1 test in 5.641s
2023-01-11T21:52:40.5526794Z 
2023-01-11T21:52:40.5526887Z OK
2023-01-11T21:52:40.5526906Z 
2023-01-11T21:52:40.5527032Z Generating XML reports...
2023-01-11T21:52:40.5527493Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213751.xml
2023-01-11T21:52:40.5527964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5528157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5528606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5528780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5528801Z 
2023-01-11T21:52:40.5528913Z Running tests...
2023-01-11T21:52:40.5529184Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5529503Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5529823Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5530048Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14300
2023-01-11T21:52:40.5530272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14301
2023-01-11T21:52:40.5530646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5530825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5531190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5531382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5531749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5531921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5532294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5532486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5532736Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5533195Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5533417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5533818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5534048Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5534278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5534515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5534758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5534992Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5535228Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5535329Z ok (5.815s)
2023-01-11T21:52:40.5535350Z 
2023-01-11T21:52:40.5535603Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5535715Z Ran 1 test in 5.815s
2023-01-11T21:52:40.5535735Z 
2023-01-11T21:52:40.5535826Z OK
2023-01-11T21:52:40.5535846Z 
2023-01-11T21:52:40.5535969Z Generating XML reports...
2023-01-11T21:52:40.5536431Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213759.xml
2023-01-11T21:52:40.5536805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5537034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5537431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5537668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5537707Z 
2023-01-11T21:52:40.5537798Z Running tests...
2023-01-11T21:52:40.5538063Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5538382Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5538698Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5538920Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14445
2023-01-11T21:52:40.5539142Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14446
2023-01-11T21:52:40.5539516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5539691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5540056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5540248Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5540620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5540798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5541176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5541369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5541620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5541867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5542253Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5542662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5542897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5543128Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5543365Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5543604Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5543842Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5544070Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5544173Z ok (5.721s)
2023-01-11T21:52:40.5544195Z 
2023-01-11T21:52:40.5544444Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5544558Z Ran 1 test in 5.721s
2023-01-11T21:52:40.5544577Z 
2023-01-11T21:52:40.5544669Z OK
2023-01-11T21:52:40.5544688Z 
2023-01-11T21:52:40.5544812Z Generating XML reports...
2023-01-11T21:52:40.5545271Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213807.xml
2023-01-11T21:52:40.5545646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5545833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5546307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5546509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5546530Z 
2023-01-11T21:52:40.5546666Z Running tests...
2023-01-11T21:52:40.5546931Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5547246Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5547623Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5547846Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14590
2023-01-11T21:52:40.5548067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14591
2023-01-11T21:52:40.5548441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5548620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5548984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5549185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5549554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5549731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5550108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5550299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5550548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5550799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5551203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5551588Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5551821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5552050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5552291Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5552527Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5552757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5552996Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5553098Z ok (5.792s)
2023-01-11T21:52:40.5553118Z 
2023-01-11T21:52:40.5553382Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5553480Z Ran 1 test in 5.792s
2023-01-11T21:52:40.5553500Z 
2023-01-11T21:52:40.5553593Z OK
2023-01-11T21:52:40.5553613Z 
2023-01-11T21:52:40.5553738Z Generating XML reports...
2023-01-11T21:52:40.5554197Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213816.xml
2023-01-11T21:52:40.5554571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5554750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5555132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5555376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5555397Z 
2023-01-11T21:52:40.5555492Z Running tests...
2023-01-11T21:52:40.5555760Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5556126Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5556505Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5556728Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14735
2023-01-11T21:52:40.5556949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14736
2023-01-11T21:52:40.5557325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5557500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5557886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5558061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5558434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5558611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5558988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5559181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5559430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5559677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5560085Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5560485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5560703Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5560932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5561169Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5561407Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5561638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5561872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5561974Z ok (5.729s)
2023-01-11T21:52:40.5561994Z 
2023-01-11T21:52:40.5562264Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5562358Z Ran 1 test in 5.729s
2023-01-11T21:52:40.5562377Z 
2023-01-11T21:52:40.5562475Z OK
2023-01-11T21:52:40.5562494Z 
2023-01-11T21:52:40.5562619Z Generating XML reports...
2023-01-11T21:52:40.5563080Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213824.xml
2023-01-11T21:52:40.5563454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5563633Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5564017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5564498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5564522Z 
2023-01-11T21:52:40.5564647Z Running tests...
2023-01-11T21:52:40.5564982Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5565317Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5565760Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5565984Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 14880
2023-01-11T21:52:40.5566202Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 14881
2023-01-11T21:52:40.5566581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5566759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5567146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5567320Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5567687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5567865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5568242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5568431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5568683Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5568932Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5569334Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5569738Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5569953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5570191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5570429Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5570658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5570884Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5571118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5571221Z ok (5.900s)
2023-01-11T21:52:40.5571241Z 
2023-01-11T21:52:40.5571509Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5571606Z Ran 1 test in 5.901s
2023-01-11T21:52:40.5571642Z 
2023-01-11T21:52:40.5571717Z OK
2023-01-11T21:52:40.5571736Z 
2023-01-11T21:52:40.5571858Z Generating XML reports...
2023-01-11T21:52:40.5572319Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213832.xml
2023-01-11T21:52:40.5572695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5572874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5573258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5573451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5573470Z 
2023-01-11T21:52:40.5573579Z Running tests...
2023-01-11T21:52:40.5573826Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5574196Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5574576Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5574839Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15025
2023-01-11T21:52:40.5575057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15026
2023-01-11T21:52:40.5575432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5575612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5575994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5576190Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5576538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5576718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5577095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5577285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5577534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5577780Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5578184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5578578Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5578811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5579027Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5579266Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5579502Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5579731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5579966Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5580068Z ok (5.892s)
2023-01-11T21:52:40.5580087Z 
2023-01-11T21:52:40.5580357Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5580469Z Ran 1 test in 5.893s
2023-01-11T21:52:40.5580489Z 
2023-01-11T21:52:40.5580563Z OK
2023-01-11T21:52:40.5580585Z 
2023-01-11T21:52:40.5580710Z Generating XML reports...
2023-01-11T21:52:40.5581170Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213841.xml
2023-01-11T21:52:40.5581549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5581724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5582105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5582298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5582318Z 
2023-01-11T21:52:40.5582425Z Running tests...
2023-01-11T21:52:40.5582667Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5582986Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5583413Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5583681Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15170
2023-01-11T21:52:40.5583903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15171
2023-01-11T21:52:40.5584280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5584460Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5584843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5585037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5585388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5585565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5585947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5586141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5586388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5586634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5587036Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5587435Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5587669Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5587880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5588122Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5588356Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5588584Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5588818Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5588919Z ok (5.749s)
2023-01-11T21:52:40.5588939Z 
2023-01-11T21:52:40.5589208Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5589319Z Ran 1 test in 5.749s
2023-01-11T21:52:40.5589339Z 
2023-01-11T21:52:40.5589413Z OK
2023-01-11T21:52:40.5589450Z 
2023-01-11T21:52:40.5589555Z Generating XML reports...
2023-01-11T21:52:40.5590013Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213849.xml
2023-01-11T21:52:40.5590392Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5590574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5590962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5591156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5591176Z 
2023-01-11T21:52:40.5591286Z Running tests...
2023-01-11T21:52:40.5591549Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5591847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5592269Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5592499Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15315
2023-01-11T21:52:40.5592759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15316
2023-01-11T21:52:40.5593135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5593315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5593701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5593893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5594264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5594427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5594804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5594997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5595246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5595493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5595898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5596300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5596532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5596767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5596986Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5597227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5597457Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5597691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5597794Z ok (5.819s)
2023-01-11T21:52:40.5597813Z 
2023-01-11T21:52:40.5598084Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5598194Z Ran 1 test in 5.819s
2023-01-11T21:52:40.5598215Z 
2023-01-11T21:52:40.5598307Z OK
2023-01-11T21:52:40.5598326Z 
2023-01-11T21:52:40.5598432Z Generating XML reports...
2023-01-11T21:52:40.5598891Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213857.xml
2023-01-11T21:52:40.5599264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5599445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5599828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5600023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5600042Z 
2023-01-11T21:52:40.5600153Z Running tests...
2023-01-11T21:52:40.5600413Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5600711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5601133Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5601364Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15460
2023-01-11T21:52:40.5601582Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15461
2023-01-11T21:52:40.5602017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5602196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5602582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5602778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5603148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5603306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5603683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5603873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5604124Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5604621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5605041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5605442Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5605678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5605907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5606132Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5606364Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5606602Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5606835Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5606936Z ok (5.700s)
2023-01-11T21:52:40.5606956Z 
2023-01-11T21:52:40.5607224Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5607334Z Ran 1 test in 5.700s
2023-01-11T21:52:40.5607354Z 
2023-01-11T21:52:40.5607447Z OK
2023-01-11T21:52:40.5607466Z 
2023-01-11T21:52:40.5607571Z Generating XML reports...
2023-01-11T21:52:40.5608028Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213906.xml
2023-01-11T21:52:40.5608408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5608587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5608978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5609173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5609193Z 
2023-01-11T21:52:40.5609301Z Running tests...
2023-01-11T21:52:40.5609563Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5609879Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5610232Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5610530Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15605
2023-01-11T21:52:40.5610758Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15606
2023-01-11T21:52:40.5611135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5611371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5611758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5611952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5612319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5612496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5612855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5613050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5613298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5613548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5613954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5614354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5614587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5614818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5615056Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5615279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5615514Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5615743Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5615845Z ok (5.709s)
2023-01-11T21:52:40.5615865Z 
2023-01-11T21:52:40.5616135Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5616247Z Ran 1 test in 5.709s
2023-01-11T21:52:40.5616267Z 
2023-01-11T21:52:40.5616357Z OK
2023-01-11T21:52:40.5616377Z 
2023-01-11T21:52:40.5616500Z Generating XML reports...
2023-01-11T21:52:40.5616938Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213914.xml
2023-01-11T21:52:40.5617313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5617493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5617878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5618075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5618094Z 
2023-01-11T21:52:40.5618205Z Running tests...
2023-01-11T21:52:40.5618469Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5618785Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5619086Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5619311Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15750
2023-01-11T21:52:40.5619530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15751
2023-01-11T21:52:40.5619951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5620135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5620562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5620755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5621117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5621291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5621655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5621844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5622095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5622341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5622748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5623149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5623380Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5623609Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5623849Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5624067Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5624309Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5624539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5624644Z ok (5.728s)
2023-01-11T21:52:40.5624664Z 
2023-01-11T21:52:40.5624930Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5625041Z Ran 1 test in 5.729s
2023-01-11T21:52:40.5625061Z 
2023-01-11T21:52:40.5625151Z OK
2023-01-11T21:52:40.5625170Z 
2023-01-11T21:52:40.5625293Z Generating XML reports...
2023-01-11T21:52:40.5625732Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213922.xml
2023-01-11T21:52:40.5626110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5626288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5626675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5626868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5626891Z 
2023-01-11T21:52:40.5627000Z Running tests...
2023-01-11T21:52:40.5627260Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5627576Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5627887Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5628090Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 15895
2023-01-11T21:52:40.5628306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 15896
2023-01-11T21:52:40.5628677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5628904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5629297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5629542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5629910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5630084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5630442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5630632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5630877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5631128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5631531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5631930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5632164Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5632397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5632635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5632898Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5633137Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5633372Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5633475Z ok (5.802s)
2023-01-11T21:52:40.5633496Z 
2023-01-11T21:52:40.5633763Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5633879Z Ran 1 test in 5.802s
2023-01-11T21:52:40.5633899Z 
2023-01-11T21:52:40.5633990Z OK
2023-01-11T21:52:40.5634009Z 
2023-01-11T21:52:40.5634132Z Generating XML reports...
2023-01-11T21:52:40.5634570Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213930.xml
2023-01-11T21:52:40.5634944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5635122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5635503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5635699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5635719Z 
2023-01-11T21:52:40.5635828Z Running tests...
2023-01-11T21:52:40.5636091Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5636410Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5636677Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5637419Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.623s)
2023-01-11T21:52:40.5637459Z 
2023-01-11T21:52:40.5637702Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5637813Z Ran 1 test in 1.623s
2023-01-11T21:52:40.5637832Z 
2023-01-11T21:52:40.5637993Z OK (skipped=1)
2023-01-11T21:52:40.5638014Z 
2023-01-11T21:52:40.5638144Z Generating XML reports...
2023-01-11T21:52:40.5638603Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213939.xml
2023-01-11T21:52:40.5639027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5639208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5639591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5639785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5639806Z 
2023-01-11T21:52:40.5639895Z Running tests...
2023-01-11T21:52:40.5640157Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5640479Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5640737Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5640961Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16074
2023-01-11T21:52:40.5641185Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16075
2023-01-11T21:52:40.5641557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5641734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5642099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5642293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5642658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5642835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5643211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5643408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5643658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5643904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5644553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5644948Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5645183Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5645416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5645517Z ok (5.705s)
2023-01-11T21:52:40.5645537Z 
2023-01-11T21:52:40.5645808Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5645921Z Ran 1 test in 5.705s
2023-01-11T21:52:40.5645941Z 
2023-01-11T21:52:40.5646036Z OK
2023-01-11T21:52:40.5646055Z 
2023-01-11T21:52:40.5646179Z Generating XML reports...
2023-01-11T21:52:40.5646619Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213943.xml
2023-01-11T21:52:40.5646996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5647177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5647558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5647825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5647847Z 
2023-01-11T21:52:40.5647964Z Running tests...
2023-01-11T21:52:40.5648286Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5648604Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5648881Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5649084Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16185
2023-01-11T21:52:40.5649304Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16186
2023-01-11T21:52:40.5649678Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5649854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5650240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5650438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5650810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5650986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5651342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5651531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5651778Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5652025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5652431Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5652831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5653068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5653299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5653539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5653757Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5654168Z /opt/conda/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmppf5o056m'>
2023-01-11T21:52:40.5654333Z   _warnings.warn(warn_message, ResourceWarning)
2023-01-11T21:52:40.5654737Z /opt/conda/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmpk04dytde'>
2023-01-11T21:52:40.5654902Z   _warnings.warn(warn_message, ResourceWarning)
2023-01-11T21:52:40.5655004Z ok (5.526s)
2023-01-11T21:52:40.5655027Z 
2023-01-11T21:52:40.5655297Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5655409Z Ran 1 test in 5.526s
2023-01-11T21:52:40.5655429Z 
2023-01-11T21:52:40.5655503Z OK
2023-01-11T21:52:40.5655540Z 
2023-01-11T21:52:40.5655646Z Generating XML reports...
2023-01-11T21:52:40.5656104Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213951.xml
2023-01-11T21:52:40.5656477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5656654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5657090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5657292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5657312Z 
2023-01-11T21:52:40.5657422Z Running tests...
2023-01-11T21:52:40.5657729Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5658027Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5658296Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5658519Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16300
2023-01-11T21:52:40.5658739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16301
2023-01-11T21:52:40.5659114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5659292Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5659675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5659866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5660215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5660390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5660764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5660953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5661202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5661449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5661856Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5662252Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5662486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5662690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5662929Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5663164Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5663267Z ok (4.227s)
2023-01-11T21:52:40.5663287Z 
2023-01-11T21:52:40.5663554Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5663666Z Ran 1 test in 4.228s
2023-01-11T21:52:40.5663686Z 
2023-01-11T21:52:40.5663778Z OK
2023-01-11T21:52:40.5663801Z 
2023-01-11T21:52:40.5663923Z Generating XML reports...
2023-01-11T21:52:40.5664378Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213959.xml
2023-01-11T21:52:40.5664739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5664918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5665305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5665497Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5665516Z 
2023-01-11T21:52:40.5665624Z Running tests...
2023-01-11T21:52:40.5665888Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5666202Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5666521Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5666734Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16443
2023-01-11T21:52:40.5667011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16444
2023-01-11T21:52:40.5667386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5667565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5667947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5668142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5668511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5668689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5669064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5669239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5669488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5669735Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5670137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5670535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5670769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5671004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5671241Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5671457Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5671561Z ok (5.645s)
2023-01-11T21:52:40.5671580Z 
2023-01-11T21:52:40.5671848Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5671960Z Ran 1 test in 5.645s
2023-01-11T21:52:40.5671980Z 
2023-01-11T21:52:40.5672071Z OK
2023-01-11T21:52:40.5672091Z 
2023-01-11T21:52:40.5672215Z Generating XML reports...
2023-01-11T21:52:40.5672668Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214006.xml
2023-01-11T21:52:40.5673040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5673221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5673584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5673780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5673799Z 
2023-01-11T21:52:40.5673906Z Running tests...
2023-01-11T21:52:40.5674170Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5674490Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5674777Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5675000Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16558
2023-01-11T21:52:40.5675220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16559
2023-01-11T21:52:40.5675622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5675807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5676244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5676436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5676801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5676981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5677358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5677549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5677802Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5678031Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5678439Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5678838Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5679071Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5679304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5679546Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.5679790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.5680193Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5680590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5680816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:52:40.5681055Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:52:40.5681445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:52:40.5681844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:52:40.5681947Z ok (5.132s)
2023-01-11T21:52:40.5681966Z 
2023-01-11T21:52:40.5682234Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5682346Z Ran 1 test in 5.132s
2023-01-11T21:52:40.5682369Z 
2023-01-11T21:52:40.5682461Z OK
2023-01-11T21:52:40.5682479Z 
2023-01-11T21:52:40.5682603Z Generating XML reports...
2023-01-11T21:52:40.5683042Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214014.xml
2023-01-11T21:52:40.5683420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5683596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5683978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5684172Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5684373Z 
2023-01-11T21:52:40.5684496Z Running tests...
2023-01-11T21:52:40.5684769Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5685161Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5685434Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5685747Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16681
2023-01-11T21:52:40.5685964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16682
2023-01-11T21:52:40.5686344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5686521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5686904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5687096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5687467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5687640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5688002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5688199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5688445Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5688690Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5689093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5689496Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5689729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5689960Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5690205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.5690429Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.5690833Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5691287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5691532Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:52:40.5691772Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:52:40.5692170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:52:40.5692569Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:52:40.5692678Z ok (15.143s)
2023-01-11T21:52:40.5692697Z 
2023-01-11T21:52:40.5692965Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5693060Z Ran 1 test in 15.143s
2023-01-11T21:52:40.5693079Z 
2023-01-11T21:52:40.5693170Z OK
2023-01-11T21:52:40.5693190Z 
2023-01-11T21:52:40.5693313Z Generating XML reports...
2023-01-11T21:52:40.5693774Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214022.xml
2023-01-11T21:52:40.5694149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5694378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5694773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5695017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5695037Z 
2023-01-11T21:52:40.5695145Z Running tests...
2023-01-11T21:52:40.5695394Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5695706Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5696018Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5696239Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16804
2023-01-11T21:52:40.5696468Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16805
2023-01-11T21:52:40.5696845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5697023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5697410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5697585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5697953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5698131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5698507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5698697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5698945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5699193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5699597Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5700000Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5700214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5700444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5700546Z ok (5.618s)
2023-01-11T21:52:40.5700565Z 
2023-01-11T21:52:40.5700830Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5700941Z Ran 1 test in 5.618s
2023-01-11T21:52:40.5700961Z 
2023-01-11T21:52:40.5701053Z OK
2023-01-11T21:52:40.5701071Z 
2023-01-11T21:52:40.5701199Z Generating XML reports...
2023-01-11T21:52:40.5701656Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214040.xml
2023-01-11T21:52:40.5702014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5702194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5702574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5702767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5702786Z 
2023-01-11T21:52:40.5702894Z Running tests...
2023-01-11T21:52:40.5703158Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5703474Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5703817Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5704045Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 16919
2023-01-11T21:52:40.5704285Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 16920
2023-01-11T21:52:40.5704658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5704837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5705217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5705408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5705776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5705953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5706329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5706502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5706749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5706993Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5707397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5707795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5708030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5708257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5708360Z ok (5.715s)
2023-01-11T21:52:40.5708379Z 
2023-01-11T21:52:40.5708646Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5708743Z Ran 1 test in 5.715s
2023-01-11T21:52:40.5708762Z 
2023-01-11T21:52:40.5708856Z OK
2023-01-11T21:52:40.5708875Z 
2023-01-11T21:52:40.5708997Z Generating XML reports...
2023-01-11T21:52:40.5709453Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214048.xml
2023-01-11T21:52:40.5709827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5710005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5710386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5710583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5710604Z 
2023-01-11T21:52:40.5710714Z Running tests...
2023-01-11T21:52:40.5710956Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5711276Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5711535Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5711757Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17034
2023-01-11T21:52:40.5711976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17035
2023-01-11T21:52:40.5712349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5712524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5712953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5713135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5713507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5713725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5714106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5714297Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5714547Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5714793Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5715198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5715599Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5715816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5716050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5716152Z ok (5.537s)
2023-01-11T21:52:40.5716172Z 
2023-01-11T21:52:40.5716438Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5716549Z Ran 1 test in 5.538s
2023-01-11T21:52:40.5716568Z 
2023-01-11T21:52:40.5716661Z OK
2023-01-11T21:52:40.5716680Z 
2023-01-11T21:52:40.5716806Z Generating XML reports...
2023-01-11T21:52:40.5717261Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214056.xml
2023-01-11T21:52:40.5717617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5717796Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5718175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5718371Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5718391Z 
2023-01-11T21:52:40.5718499Z Running tests...
2023-01-11T21:52:40.5718761Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5719075Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5719341Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5719563Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17145
2023-01-11T21:52:40.5719766Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17146
2023-01-11T21:52:40.5720140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5720319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5720702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5720893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5721255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5721431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5721807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5721978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5722274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5722530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5722984Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5723375Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5723611Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5723840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5724867Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:52:40.5725666Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:52:40.5725908Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5726151Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5726253Z ok (5.543s)
2023-01-11T21:52:40.5726274Z 
2023-01-11T21:52:40.5726552Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5726653Z Ran 1 test in 5.543s
2023-01-11T21:52:40.5726672Z 
2023-01-11T21:52:40.5726767Z OK
2023-01-11T21:52:40.5726787Z 
2023-01-11T21:52:40.5726910Z Generating XML reports...
2023-01-11T21:52:40.5727372Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214104.xml
2023-01-11T21:52:40.5727749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5727928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5728309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5728506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5728526Z 
2023-01-11T21:52:40.5728617Z Running tests...
2023-01-11T21:52:40.5728880Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5729196Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5729480Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5730236Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.596s)
2023-01-11T21:52:40.5730257Z 
2023-01-11T21:52:40.5730518Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5730629Z Ran 1 test in 1.597s
2023-01-11T21:52:40.5730725Z 
2023-01-11T21:52:40.5730840Z OK (skipped=1)
2023-01-11T21:52:40.5730860Z 
2023-01-11T21:52:40.5730985Z Generating XML reports...
2023-01-11T21:52:40.5731512Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214112.xml
2023-01-11T21:52:40.5731870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5732049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5732434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5732629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5732648Z 
2023-01-11T21:52:40.5732758Z Running tests...
2023-01-11T21:52:40.5733065Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5733391Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5733676Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5734430Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.593s)
2023-01-11T21:52:40.5734451Z 
2023-01-11T21:52:40.5734712Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5734807Z Ran 1 test in 1.593s
2023-01-11T21:52:40.5734827Z 
2023-01-11T21:52:40.5734934Z OK (skipped=1)
2023-01-11T21:52:40.5734953Z 
2023-01-11T21:52:40.5735076Z Generating XML reports...
2023-01-11T21:52:40.5735534Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214116.xml
2023-01-11T21:52:40.5735908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5736094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5736474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5736667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5736687Z 
2023-01-11T21:52:40.5736777Z Running tests...
2023-01-11T21:52:40.5737041Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5737355Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5737637Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5737863Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17328
2023-01-11T21:52:40.5738083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17329
2023-01-11T21:52:40.5738458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5738642Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5739024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5739197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5739574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5739749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5740172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5740373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5740628Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5740920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5741330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5741714Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5741948Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5742180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5742526Z STAGE:2023-01-11 21:41:25 17328:17328 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5742860Z STAGE:2023-01-11 21:41:25 17329:17329 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5743100Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5743341Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5743679Z STAGE:2023-01-11 21:41:26 17329:17329 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5744010Z STAGE:2023-01-11 21:41:26 17328:17328 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5744342Z STAGE:2023-01-11 21:41:26 17329:17329 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5744695Z STAGE:2023-01-11 21:41:26 17328:17328 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5745491Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:52:40.5746283Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:52:40.5746629Z STAGE:2023-01-11 21:41:26 17328:17328 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5746953Z STAGE:2023-01-11 21:41:26 17329:17329 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5747293Z STAGE:2023-01-11 21:41:26 17328:17328 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5747622Z STAGE:2023-01-11 21:41:26 17329:17329 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5747970Z STAGE:2023-01-11 21:41:26 17328:17328 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5748318Z STAGE:2023-01-11 21:41:26 17329:17329 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5748420Z ok (5.940s)
2023-01-11T21:52:40.5748440Z 
2023-01-11T21:52:40.5748705Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5748799Z Ran 1 test in 5.940s
2023-01-11T21:52:40.5748837Z 
2023-01-11T21:52:40.5748963Z OK
2023-01-11T21:52:40.5748985Z 
2023-01-11T21:52:40.5749113Z Generating XML reports...
2023-01-11T21:52:40.5749573Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214120.xml
2023-01-11T21:52:40.5749997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5750175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5750560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5750755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5750775Z 
2023-01-11T21:52:40.5750883Z Running tests...
2023-01-11T21:52:40.5751127Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5751449Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5751718Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5751946Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17447
2023-01-11T21:52:40.5752167Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17448
2023-01-11T21:52:40.5752543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5752719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5753102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5753295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5753644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5753821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5754201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5754393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5754642Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5754891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5755294Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5755694Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5755909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5756146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5756250Z ok (5.141s)
2023-01-11T21:52:40.5756272Z 
2023-01-11T21:52:40.5756541Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5756652Z Ran 1 test in 5.141s
2023-01-11T21:52:40.5756671Z 
2023-01-11T21:52:40.5756762Z OK
2023-01-11T21:52:40.5756781Z 
2023-01-11T21:52:40.5756904Z Generating XML reports...
2023-01-11T21:52:40.5757359Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214129.xml
2023-01-11T21:52:40.5757738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5757899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5758331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5758530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5758549Z 
2023-01-11T21:52:40.5758659Z Running tests...
2023-01-11T21:52:40.5758969Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5759286Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5759567Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5760320Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.629s)
2023-01-11T21:52:40.5760340Z 
2023-01-11T21:52:40.5760603Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5760701Z Ran 1 test in 1.629s
2023-01-11T21:52:40.5760739Z 
2023-01-11T21:52:40.5760829Z OK (skipped=1)
2023-01-11T21:52:40.5760848Z 
2023-01-11T21:52:40.5760973Z Generating XML reports...
2023-01-11T21:52:40.5761430Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214136.xml
2023-01-11T21:52:40.5761806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5761986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5762373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5762565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5762585Z 
2023-01-11T21:52:40.5762692Z Running tests...
2023-01-11T21:52:40.5762939Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5763259Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5763544Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5763769Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17592
2023-01-11T21:52:40.5763990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17593
2023-01-11T21:52:40.5764617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5764805Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5765193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5765388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5765745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5765918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5766299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5766490Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5766739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5766985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5767389Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5767784Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5768072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5768310Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5769288Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T21:52:40.5769406Z   warnings.warn(
2023-01-11T21:52:40.5770327Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T21:52:40.5770440Z   warnings.warn(
2023-01-11T21:52:40.5770678Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5770913Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5771014Z ok (5.520s)
2023-01-11T21:52:40.5771034Z 
2023-01-11T21:52:40.5771299Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5771393Z Ran 1 test in 5.521s
2023-01-11T21:52:40.5771431Z 
2023-01-11T21:52:40.5771505Z OK
2023-01-11T21:52:40.5771524Z 
2023-01-11T21:52:40.5771646Z Generating XML reports...
2023-01-11T21:52:40.5772101Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214141.xml
2023-01-11T21:52:40.5772474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5772653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5773039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5773235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5773255Z 
2023-01-11T21:52:40.5773363Z Running tests...
2023-01-11T21:52:40.5773607Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5773927Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5774205Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5774959Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.620s)
2023-01-11T21:52:40.5774980Z 
2023-01-11T21:52:40.5775244Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5775358Z Ran 1 test in 1.620s
2023-01-11T21:52:40.5775378Z 
2023-01-11T21:52:40.5775488Z OK (skipped=1)
2023-01-11T21:52:40.5775507Z 
2023-01-11T21:52:40.5775629Z Generating XML reports...
2023-01-11T21:52:40.5776087Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214149.xml
2023-01-11T21:52:40.5776463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5776623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5777007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5777250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5777272Z 
2023-01-11T21:52:40.5777387Z Running tests...
2023-01-11T21:52:40.5777655Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5778019Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5778296Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5778517Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17741
2023-01-11T21:52:40.5778717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17742
2023-01-11T21:52:40.5779093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5779269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5779657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5779850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5780219Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5780396Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5780772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5780960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5781192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5781438Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5781844Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5782245Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5782482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5782706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5783047Z STAGE:2023-01-11 21:41:58 17742:17742 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5783372Z STAGE:2023-01-11 21:41:58 17741:17741 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5783612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5783831Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T21:52:40.5784386Z STAGE:2023-01-11 21:41:58 17741:17741 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:41:58 17742:17742 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5784408Z 
2023-01-11T21:52:40.5784766Z STAGE:2023-01-11 21:41:58 17741:17741 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5785116Z STAGE:2023-01-11 21:41:58 17742:17742 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5785445Z STAGE:2023-01-11 21:41:58 17741:17741 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5785785Z STAGE:2023-01-11 21:41:58 17741:17741 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5786133Z STAGE:2023-01-11 21:41:58 17741:17741 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5786235Z ok (6.135s)
2023-01-11T21:52:40.5786254Z 
2023-01-11T21:52:40.5786521Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5786665Z Ran 1 test in 6.136s
2023-01-11T21:52:40.5786705Z 
2023-01-11T21:52:40.5786784Z OK
2023-01-11T21:52:40.5786803Z 
2023-01-11T21:52:40.5786930Z Generating XML reports...
2023-01-11T21:52:40.5787432Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214153.xml
2023-01-11T21:52:40.5787808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5787989Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5788372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5788567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5788587Z 
2023-01-11T21:52:40.5788696Z Running tests...
2023-01-11T21:52:40.5788939Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5789258Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5789527Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5789752Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17860
2023-01-11T21:52:40.5789973Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17861
2023-01-11T21:52:40.5790347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5790526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5790908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5791082Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5791453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5791627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5792007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5792201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5792448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5792695Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5793098Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5793499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5793715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5793945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5794050Z ok (5.018s)
2023-01-11T21:52:40.5794070Z 
2023-01-11T21:52:40.5794337Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5794448Z Ran 1 test in 5.019s
2023-01-11T21:52:40.5794467Z 
2023-01-11T21:52:40.5794561Z OK
2023-01-11T21:52:40.5794580Z 
2023-01-11T21:52:40.5794703Z Generating XML reports...
2023-01-11T21:52:40.5795158Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214202.xml
2023-01-11T21:52:40.5795513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5795693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5796122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5796323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5796392Z 
2023-01-11T21:52:40.5796506Z Running tests...
2023-01-11T21:52:40.5796774Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5797093Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5797369Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5797592Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 17971
2023-01-11T21:52:40.5797793Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 17972
2023-01-11T21:52:40.5798167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5798347Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5798730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5798924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5799292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5799464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5799841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5800010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5800258Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5800507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5800908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5801309Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5801544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5801774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5801876Z ok (5.048s)
2023-01-11T21:52:40.5801895Z 
2023-01-11T21:52:40.5802161Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5802256Z Ran 1 test in 5.048s
2023-01-11T21:52:40.5802275Z 
2023-01-11T21:52:40.5802371Z OK
2023-01-11T21:52:40.5802391Z 
2023-01-11T21:52:40.5802513Z Generating XML reports...
2023-01-11T21:52:40.5802969Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214209.xml
2023-01-11T21:52:40.5803345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5803529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5803912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5804105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5804124Z 
2023-01-11T21:52:40.5804454Z Running tests...
2023-01-11T21:52:40.5804716Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5805037Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5805314Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5806141Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.622s)
2023-01-11T21:52:40.5806206Z 
2023-01-11T21:52:40.5806479Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5806592Z Ran 1 test in 1.622s
2023-01-11T21:52:40.5806611Z 
2023-01-11T21:52:40.5806719Z OK (skipped=1)
2023-01-11T21:52:40.5806739Z 
2023-01-11T21:52:40.5806863Z Generating XML reports...
2023-01-11T21:52:40.5807320Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214217.xml
2023-01-11T21:52:40.5807676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5807862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5808246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5808444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5808464Z 
2023-01-11T21:52:40.5808571Z Running tests...
2023-01-11T21:52:40.5808837Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5809153Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5809414Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5810160Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.640s)
2023-01-11T21:52:40.5810184Z 
2023-01-11T21:52:40.5810450Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5810544Z Ran 1 test in 1.640s
2023-01-11T21:52:40.5810566Z 
2023-01-11T21:52:40.5810673Z OK (skipped=1)
2023-01-11T21:52:40.5810692Z 
2023-01-11T21:52:40.5810815Z Generating XML reports...
2023-01-11T21:52:40.5811270Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214221.xml
2023-01-11T21:52:40.5811645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5811825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5812210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5812402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5812422Z 
2023-01-11T21:52:40.5812533Z Running tests...
2023-01-11T21:52:40.5812778Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5813095Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5813392Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5814137Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.641s)
2023-01-11T21:52:40.5814157Z 
2023-01-11T21:52:40.5814416Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5814528Z Ran 1 test in 1.642s
2023-01-11T21:52:40.5814547Z 
2023-01-11T21:52:40.5814654Z OK (skipped=1)
2023-01-11T21:52:40.5814672Z 
2023-01-11T21:52:40.5814842Z Generating XML reports...
2023-01-11T21:52:40.5815307Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214225.xml
2023-01-11T21:52:40.5815710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5815888Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5816267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5816458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5816477Z 
2023-01-11T21:52:40.5816587Z Running tests...
2023-01-11T21:52:40.5816850Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5817166Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5817469Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5817692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18184
2023-01-11T21:52:40.5817898Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18185
2023-01-11T21:52:40.5818270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5818446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5818826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5819019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5819390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5819571Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5819949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5820124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5820374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5820619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5821025Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5821424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5821658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5821892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5821995Z ok (5.638s)
2023-01-11T21:52:40.5822015Z 
2023-01-11T21:52:40.5822282Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5822376Z Ran 1 test in 5.638s
2023-01-11T21:52:40.5822396Z 
2023-01-11T21:52:40.5822491Z OK
2023-01-11T21:52:40.5822510Z 
2023-01-11T21:52:40.5822633Z Generating XML reports...
2023-01-11T21:52:40.5823087Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214229.xml
2023-01-11T21:52:40.5823462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5823640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5824021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5824263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5824286Z 
2023-01-11T21:52:40.5824401Z Running tests...
2023-01-11T21:52:40.5824648Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5825012Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5825287Z   test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5825513Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18299
2023-01-11T21:52:40.5825736Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18300
2023-01-11T21:52:40.5826108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5826284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5826671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5826843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5827213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5827388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5827763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5827953Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5828203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5828450Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5828855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5829254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5829470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5829702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5830085Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2023-01-11T21:52:40.5830342Z   warnings.warn("Initializing zero-element tensors is a no-op")
2023-01-11T21:52:40.5830716Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2023-01-11T21:52:40.5830969Z   warnings.warn("Initializing zero-element tensors is a no-op")
2023-01-11T21:52:40.5831074Z ok (5.023s)
2023-01-11T21:52:40.5831093Z 
2023-01-11T21:52:40.5831357Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5831453Z Ran 1 test in 5.023s
2023-01-11T21:52:40.5831490Z 
2023-01-11T21:52:40.5831567Z OK
2023-01-11T21:52:40.5831586Z 
2023-01-11T21:52:40.5831709Z Generating XML reports...
2023-01-11T21:52:40.5832163Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214237.xml
2023-01-11T21:52:40.5832534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5832711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5833144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5833340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5833359Z 
2023-01-11T21:52:40.5833469Z Running tests...
2023-01-11T21:52:40.5833807Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5834136Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5834441Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5834663Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18410
2023-01-11T21:52:40.5834883Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18411
2023-01-11T21:52:40.5835261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5835437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5835820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5835997Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5836368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5836548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5836921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5837113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5837361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5837612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5838012Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5838414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5838628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5838861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5839102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.5839346Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.5839750Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5840145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5840249Z ok (4.229s)
2023-01-11T21:52:40.5840270Z 
2023-01-11T21:52:40.5840531Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5840646Z Ran 1 test in 4.229s
2023-01-11T21:52:40.5840666Z 
2023-01-11T21:52:40.5840740Z OK
2023-01-11T21:52:40.5840759Z 
2023-01-11T21:52:40.5840884Z Generating XML reports...
2023-01-11T21:52:40.5841340Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214245.xml
2023-01-11T21:52:40.5841715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5841892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5842273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5842467Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5842488Z 
2023-01-11T21:52:40.5842595Z Running tests...
2023-01-11T21:52:40.5842837Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5843206Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5843469Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5843737Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18525
2023-01-11T21:52:40.5843960Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18526
2023-01-11T21:52:40.5844575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5844761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5845148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5845342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5845694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5845872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5846256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5846446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5846697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5846949Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5847352Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5847750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5847987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5848202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5848448Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.5848693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.5849094Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5849491Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5849593Z ok (4.226s)
2023-01-11T21:52:40.5849613Z 
2023-01-11T21:52:40.5849877Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5849988Z Ran 1 test in 4.227s
2023-01-11T21:52:40.5850008Z 
2023-01-11T21:52:40.5850086Z OK
2023-01-11T21:52:40.5850122Z 
2023-01-11T21:52:40.5850228Z Generating XML reports...
2023-01-11T21:52:40.5850679Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214252.xml
2023-01-11T21:52:40.5851057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5851235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5851618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5851812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5851832Z 
2023-01-11T21:52:40.5851941Z Running tests...
2023-01-11T21:52:40.5852204Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5852575Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5852866Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5853684Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.595s)
2023-01-11T21:52:40.5853706Z 
2023-01-11T21:52:40.5853964Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5854076Z Ran 1 test in 1.595s
2023-01-11T21:52:40.5854095Z 
2023-01-11T21:52:40.5854203Z OK (skipped=1)
2023-01-11T21:52:40.5854222Z 
2023-01-11T21:52:40.5854351Z Generating XML reports...
2023-01-11T21:52:40.5854806Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214258.xml
2023-01-11T21:52:40.5855181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5855359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5855728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5855922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5855941Z 
2023-01-11T21:52:40.5856049Z Running tests...
2023-01-11T21:52:40.5856313Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5856631Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5856911Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5857658Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.592s)
2023-01-11T21:52:40.5857681Z 
2023-01-11T21:52:40.5857943Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5858054Z Ran 1 test in 1.593s
2023-01-11T21:52:40.5858073Z 
2023-01-11T21:52:40.5858162Z OK (skipped=1)
2023-01-11T21:52:40.5858199Z 
2023-01-11T21:52:40.5858305Z Generating XML reports...
2023-01-11T21:52:40.5858756Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214303.xml
2023-01-11T21:52:40.5859131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5859310Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5859696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5859888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5859910Z 
2023-01-11T21:52:40.5860018Z Running tests...
2023-01-11T21:52:40.5860282Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5860581Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5860856Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5861077Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18708
2023-01-11T21:52:40.5861298Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18709
2023-01-11T21:52:40.5861674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5861907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5862306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5862562Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5862914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5863091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5863469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5863663Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5863913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5864161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5864577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5864980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5865213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5865426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5865530Z ok (4.258s)
2023-01-11T21:52:40.5865550Z 
2023-01-11T21:52:40.5865815Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5865927Z Ran 1 test in 4.258s
2023-01-11T21:52:40.5865946Z 
2023-01-11T21:52:40.5866037Z OK
2023-01-11T21:52:40.5866056Z 
2023-01-11T21:52:40.5866181Z Generating XML reports...
2023-01-11T21:52:40.5866640Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214307.xml
2023-01-11T21:52:40.5867016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5867200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5867564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5867758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5867777Z 
2023-01-11T21:52:40.5867886Z Running tests...
2023-01-11T21:52:40.5868150Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5868467Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5868710Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5868937Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18817
2023-01-11T21:52:40.5869157Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18818
2023-01-11T21:52:40.5869511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5869692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5870074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5870268Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5870636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5870814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5871191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5871432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5871689Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5871958Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5872367Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5872769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5873005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5873235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5873581Z STAGE:2023-01-11 21:43:17 18817:18817 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5873908Z STAGE:2023-01-11 21:43:17 18818:18818 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5874247Z STAGE:2023-01-11 21:43:17 18818:18818 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5874601Z STAGE:2023-01-11 21:43:17 18818:18818 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5874919Z STAGE:2023-01-11 21:43:17 18817:18817 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5875269Z STAGE:2023-01-11 21:43:17 18817:18817 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5875602Z STAGE:2023-01-11 21:43:17 18818:18818 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5875926Z STAGE:2023-01-11 21:43:17 18817:18817 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5876262Z STAGE:2023-01-11 21:43:17 18817:18817 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5876611Z STAGE:2023-01-11 21:43:17 18817:18817 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5876948Z STAGE:2023-01-11 21:43:17 18818:18818 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5877295Z STAGE:2023-01-11 21:43:17 18818:18818 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5877380Z ok (4.104s)
2023-01-11T21:52:40.5877418Z 
2023-01-11T21:52:40.5877666Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5877780Z Ran 1 test in 4.104s
2023-01-11T21:52:40.5877799Z 
2023-01-11T21:52:40.5877891Z OK
2023-01-11T21:52:40.5877910Z 
2023-01-11T21:52:40.5878034Z Generating XML reports...
2023-01-11T21:52:40.5878487Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214313.xml
2023-01-11T21:52:40.5878863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5879044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5879429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5879605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5879624Z 
2023-01-11T21:52:40.5879733Z Running tests...
2023-01-11T21:52:40.5879997Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5880317Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5880573Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5880794Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 18930
2023-01-11T21:52:40.5881063Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 18931
2023-01-11T21:52:40.5881451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5881656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5882046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5882241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5882608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5882784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5883163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5883356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5883607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5883855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5884464Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5884880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5885113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5885345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5885446Z ok (4.230s)
2023-01-11T21:52:40.5885466Z 
2023-01-11T21:52:40.5885735Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5885850Z Ran 1 test in 4.231s
2023-01-11T21:52:40.5885875Z 
2023-01-11T21:52:40.5885967Z OK
2023-01-11T21:52:40.5885987Z 
2023-01-11T21:52:40.5886109Z Generating XML reports...
2023-01-11T21:52:40.5886548Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214320.xml
2023-01-11T21:52:40.5886930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5887105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5887486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5887678Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5887698Z 
2023-01-11T21:52:40.5887805Z Running tests...
2023-01-11T21:52:40.5888067Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5888387Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5888626Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2023-01-11T21:52:40.5888667Z 
2023-01-11T21:52:40.5888912Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5889023Z Ran 1 test in 0.002s
2023-01-11T21:52:40.5889042Z 
2023-01-11T21:52:40.5889149Z OK (skipped=1)
2023-01-11T21:52:40.5889167Z 
2023-01-11T21:52:40.5889290Z Generating XML reports...
2023-01-11T21:52:40.5889741Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214327.xml
2023-01-11T21:52:40.5890114Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5890290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5890748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5890931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5890998Z 
2023-01-11T21:52:40.5891114Z Running tests...
2023-01-11T21:52:40.5891377Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5891691Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5891954Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5892176Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19072
2023-01-11T21:52:40.5892395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19073
2023-01-11T21:52:40.5892768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5892930Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5893313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5893512Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5893878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5894055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5894431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5894620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5894868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5895115Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5895504Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5895905Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5896139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5896369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5896612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.5896854Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.5897252Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5897648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5897989Z STAGE:2023-01-11 21:43:33 19073:19073 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5898300Z STAGE:2023-01-11 21:43:33 19072:19072 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5898639Z STAGE:2023-01-11 21:43:33 19073:19073 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5898989Z STAGE:2023-01-11 21:43:33 19073:19073 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5899324Z STAGE:2023-01-11 21:43:33 19072:19072 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5899673Z STAGE:2023-01-11 21:43:33 19072:19072 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5900004Z STAGE:2023-01-11 21:43:33 19073:19073 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5900386Z STAGE:2023-01-11 21:43:33 19072:19072 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5900732Z STAGE:2023-01-11 21:43:33 19072:19072 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5901125Z STAGE:2023-01-11 21:43:33 19072:19072 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5901442Z STAGE:2023-01-11 21:43:33 19073:19073 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5901788Z STAGE:2023-01-11 21:43:33 19073:19073 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5901891Z ok (4.246s)
2023-01-11T21:52:40.5901910Z 
2023-01-11T21:52:40.5902177Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5902289Z Ran 1 test in 4.246s
2023-01-11T21:52:40.5902309Z 
2023-01-11T21:52:40.5902401Z OK
2023-01-11T21:52:40.5902420Z 
2023-01-11T21:52:40.5902545Z Generating XML reports...
2023-01-11T21:52:40.5903004Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214329.xml
2023-01-11T21:52:40.5903376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5903539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5903921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5904116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5904135Z 
2023-01-11T21:52:40.5904243Z Running tests...
2023-01-11T21:52:40.5904508Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5904825Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5905082Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5905304Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19191
2023-01-11T21:52:40.5905507Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19192
2023-01-11T21:52:40.5905886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5906062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5906447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5906639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5907011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5907187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5907566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5907755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5907987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5908231Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5908636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5909035Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5909269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5909498Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5909705Z skip: Skipped due to small world size. (4.144s)
2023-01-11T21:52:40.5909727Z 
2023-01-11T21:52:40.5910002Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5910140Z Ran 1 test in 4.144s
2023-01-11T21:52:40.5910180Z 
2023-01-11T21:52:40.5910270Z OK (skipped=1)
2023-01-11T21:52:40.5910288Z 
2023-01-11T21:52:40.5910412Z Generating XML reports...
2023-01-11T21:52:40.5910872Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214336.xml
2023-01-11T21:52:40.5911245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5911425Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5911810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5912007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5912026Z 
2023-01-11T21:52:40.5912134Z Running tests...
2023-01-11T21:52:40.5912380Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5912701Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5912953Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5913174Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19300
2023-01-11T21:52:40.5913391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19301
2023-01-11T21:52:40.5913764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5913941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5914325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5914499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5914867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5915044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5915419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5915605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5915855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5916102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5916500Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5916902Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5917117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5917355Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5917457Z ok (4.218s)
2023-01-11T21:52:40.5917477Z 
2023-01-11T21:52:40.5917744Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5917854Z Ran 1 test in 4.218s
2023-01-11T21:52:40.5917873Z 
2023-01-11T21:52:40.5917964Z OK
2023-01-11T21:52:40.5917983Z 
2023-01-11T21:52:40.5918107Z Generating XML reports...
2023-01-11T21:52:40.5918560Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214343.xml
2023-01-11T21:52:40.5918984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5919150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5919537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5919781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5919800Z 
2023-01-11T21:52:40.5919909Z Running tests...
2023-01-11T21:52:40.5920174Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5920492Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5920766Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5921528Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.621s)
2023-01-11T21:52:40.5921549Z 
2023-01-11T21:52:40.5921811Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5921909Z Ran 1 test in 1.621s
2023-01-11T21:52:40.5921945Z 
2023-01-11T21:52:40.5922035Z OK (skipped=1)
2023-01-11T21:52:40.5922054Z 
2023-01-11T21:52:40.5922180Z Generating XML reports...
2023-01-11T21:52:40.5922635Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214349.xml
2023-01-11T21:52:40.5923008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5923185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5923571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5923765Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5923786Z 
2023-01-11T21:52:40.5923895Z Running tests...
2023-01-11T21:52:40.5924140Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5924697Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5924952Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5925175Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19443
2023-01-11T21:52:40.5925398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19444
2023-01-11T21:52:40.5925775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5925952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5926335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5926510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5926884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5927057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5927432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5927624Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5927873Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5928120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5928599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5929013Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5929298Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5929531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5929775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.5930024Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.5930427Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5930822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5930929Z ok (4.350s)
2023-01-11T21:52:40.5930948Z 
2023-01-11T21:52:40.5931213Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5931326Z Ran 1 test in 4.350s
2023-01-11T21:52:40.5931346Z 
2023-01-11T21:52:40.5931420Z OK
2023-01-11T21:52:40.5931439Z 
2023-01-11T21:52:40.5931563Z Generating XML reports...
2023-01-11T21:52:40.5932022Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214354.xml
2023-01-11T21:52:40.5932397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5932580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5933003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5933201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5933221Z 
2023-01-11T21:52:40.5933333Z Running tests...
2023-01-11T21:52:40.5933580Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5933897Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5934149Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5934369Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19558
2023-01-11T21:52:40.5934588Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19559
2023-01-11T21:52:40.5934960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5935138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5935515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5935708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5936053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5936230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5936613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5936803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5937050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5937295Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5937699Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5938153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5938392Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5938649Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5938754Z ok (4.222s)
2023-01-11T21:52:40.5938773Z 
2023-01-11T21:52:40.5939039Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5939148Z Ran 1 test in 4.222s
2023-01-11T21:52:40.5939168Z 
2023-01-11T21:52:40.5939260Z OK
2023-01-11T21:52:40.5939280Z 
2023-01-11T21:52:40.5939401Z Generating XML reports...
2023-01-11T21:52:40.5939858Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214400.xml
2023-01-11T21:52:40.5940221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5940384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5940755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5940942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5940961Z 
2023-01-11T21:52:40.5941060Z Running tests...
2023-01-11T21:52:40.5941311Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5941616Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5941854Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5942065Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19667
2023-01-11T21:52:40.5942264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19668
2023-01-11T21:52:40.5942643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5942821Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5943205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5943549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5943973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5944151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5944529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5944720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5944954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5945202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5945713Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5946122Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5946358Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5946590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5946692Z ok (4.435s)
2023-01-11T21:52:40.5946712Z 
2023-01-11T21:52:40.5946978Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5947088Z Ran 1 test in 4.435s
2023-01-11T21:52:40.5947109Z 
2023-01-11T21:52:40.5947184Z OK
2023-01-11T21:52:40.5947203Z 
2023-01-11T21:52:40.5947389Z Generating XML reports...
2023-01-11T21:52:40.5947859Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214407.xml
2023-01-11T21:52:40.5948284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5948464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5948848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5949044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5949063Z 
2023-01-11T21:52:40.5949170Z Running tests...
2023-01-11T21:52:40.5949416Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5949734Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5950010Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5950233Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19776
2023-01-11T21:52:40.5950457Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19777
2023-01-11T21:52:40.5950830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5951006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5951387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5951579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5951927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5952106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5952485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5952679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5952929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5953176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5953577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5953977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5954209Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5954424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5954667Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.5954915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.5955315Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5955712Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5955814Z ok (4.205s)
2023-01-11T21:52:40.5955834Z 
2023-01-11T21:52:40.5956099Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5956211Z Ran 1 test in 4.205s
2023-01-11T21:52:40.5956231Z 
2023-01-11T21:52:40.5956306Z OK
2023-01-11T21:52:40.5956342Z 
2023-01-11T21:52:40.5956449Z Generating XML reports...
2023-01-11T21:52:40.5956954Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214414.xml
2023-01-11T21:52:40.5957341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5957564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5957951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5958146Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5958166Z 
2023-01-11T21:52:40.5958273Z Running tests...
2023-01-11T21:52:40.5958537Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5958835Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5959100Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5959322Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 19891
2023-01-11T21:52:40.5959541Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 19892
2023-01-11T21:52:40.5959920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5960099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5960484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5960677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5961026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5961202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5961581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5961773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5962025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5962272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5962675Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5963069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5963301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5963512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5963756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.5963998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.5964586Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5964999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.5965103Z ok (4.233s)
2023-01-11T21:52:40.5965123Z 
2023-01-11T21:52:40.5965389Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5965502Z Ran 1 test in 4.233s
2023-01-11T21:52:40.5965522Z 
2023-01-11T21:52:40.5965615Z OK
2023-01-11T21:52:40.5965634Z 
2023-01-11T21:52:40.5965741Z Generating XML reports...
2023-01-11T21:52:40.5966270Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214421.xml
2023-01-11T21:52:40.5966656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5966893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5967282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5967477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5967497Z 
2023-01-11T21:52:40.5967606Z Running tests...
2023-01-11T21:52:40.5967869Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5968167Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5968434Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5968659Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20006
2023-01-11T21:52:40.5968881Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20007
2023-01-11T21:52:40.5969257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5969440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5969819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5970010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5970378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5970534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5970915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5971106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5971354Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5971600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5972007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5972406Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5972637Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5972867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5972950Z ok (5.572s)
2023-01-11T21:52:40.5972970Z 
2023-01-11T21:52:40.5973237Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5973351Z Ran 1 test in 5.573s
2023-01-11T21:52:40.5973370Z 
2023-01-11T21:52:40.5973463Z OK
2023-01-11T21:52:40.5973482Z 
2023-01-11T21:52:40.5973608Z Generating XML reports...
2023-01-11T21:52:40.5974063Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214428.xml
2023-01-11T21:52:40.5974437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5974614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5974977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5975170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5975189Z 
2023-01-11T21:52:40.5975300Z Running tests...
2023-01-11T21:52:40.5975610Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5975941Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5976269Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5976491Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20121
2023-01-11T21:52:40.5976712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20122
2023-01-11T21:52:40.5977087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5977243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5977629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5977822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5978196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5978374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5978754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5978942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5979192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5979417Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5979822Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5980219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5980456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5980688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5980794Z ok (4.253s)
2023-01-11T21:52:40.5980813Z 
2023-01-11T21:52:40.5981078Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5981190Z Ran 1 test in 4.253s
2023-01-11T21:52:40.5981209Z 
2023-01-11T21:52:40.5981306Z OK
2023-01-11T21:52:40.5981325Z 
2023-01-11T21:52:40.5981430Z Generating XML reports...
2023-01-11T21:52:40.5981887Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214436.xml
2023-01-11T21:52:40.5982265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5982445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5982829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5983021Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5983044Z 
2023-01-11T21:52:40.5983152Z Running tests...
2023-01-11T21:52:40.5983414Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5983711Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5983953Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5984169Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20230
2023-01-11T21:52:40.5984389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20231
2023-01-11T21:52:40.5984759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5984986Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5985378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5985616Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5985982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5986140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5986514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5986703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5986950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5987202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5987604Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5988006Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5988239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5988470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5988554Z ok (4.340s)
2023-01-11T21:52:40.5988574Z 
2023-01-11T21:52:40.5988835Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5988945Z Ran 1 test in 4.341s
2023-01-11T21:52:40.5988966Z 
2023-01-11T21:52:40.5989057Z OK
2023-01-11T21:52:40.5989076Z 
2023-01-11T21:52:40.5989200Z Generating XML reports...
2023-01-11T21:52:40.5989656Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214443.xml
2023-01-11T21:52:40.5990028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5990210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5990583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5990777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5990796Z 
2023-01-11T21:52:40.5990906Z Running tests...
2023-01-11T21:52:40.5991225Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5991551Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.5991824Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.5992049Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20339
2023-01-11T21:52:40.5992269Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20340
2023-01-11T21:52:40.5992625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5992802Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5993180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5993376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5993740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.5993916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.5994345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.5994545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.5994796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.5995081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.5995489Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5995889Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.5996124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.5996356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.5996701Z STAGE:2023-01-11 21:44:53 20339:20339 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5997028Z STAGE:2023-01-11 21:44:53 20340:20340 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.5997370Z STAGE:2023-01-11 21:44:53 20340:20340 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5997723Z STAGE:2023-01-11 21:44:53 20340:20340 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5998039Z STAGE:2023-01-11 21:44:53 20339:20339 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.5998387Z STAGE:2023-01-11 21:44:53 20339:20339 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.5998490Z ok (4.328s)
2023-01-11T21:52:40.5998509Z 
2023-01-11T21:52:40.5998782Z ----------------------------------------------------------------------
2023-01-11T21:52:40.5998893Z Ran 1 test in 4.328s
2023-01-11T21:52:40.5998913Z 
2023-01-11T21:52:40.5999005Z OK
2023-01-11T21:52:40.5999028Z 
2023-01-11T21:52:40.5999154Z Generating XML reports...
2023-01-11T21:52:40.5999615Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214450.xml
2023-01-11T21:52:40.5999973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6000152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6000531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6000726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6000745Z 
2023-01-11T21:52:40.6000854Z Running tests...
2023-01-11T21:52:40.6001116Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6001433Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6001702Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6001925Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20452
2023-01-11T21:52:40.6002131Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20453
2023-01-11T21:52:40.6002504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6002678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6003058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6003250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6003617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6003842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6004399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6004654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6004906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6005155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6005566Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6005964Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6006197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6006434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6006773Z STAGE:2023-01-11 21:45:00 20452:20452 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6007104Z STAGE:2023-01-11 21:45:00 20453:20453 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6007422Z STAGE:2023-01-11 21:45:00 20453:20453 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6007775Z STAGE:2023-01-11 21:45:00 20453:20453 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6008115Z STAGE:2023-01-11 21:45:00 20452:20452 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6008463Z STAGE:2023-01-11 21:45:00 20452:20452 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6008563Z ok (4.221s)
2023-01-11T21:52:40.6008583Z 
2023-01-11T21:52:40.6008849Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6008964Z Ran 1 test in 4.221s
2023-01-11T21:52:40.6008984Z 
2023-01-11T21:52:40.6009076Z OK
2023-01-11T21:52:40.6009095Z 
2023-01-11T21:52:40.6009219Z Generating XML reports...
2023-01-11T21:52:40.6009663Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214456.xml
2023-01-11T21:52:40.6010039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6010215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6010598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6010791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6010810Z 
2023-01-11T21:52:40.6010919Z Running tests...
2023-01-11T21:52:40.6011181Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6011500Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6011768Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6011999Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20565
2023-01-11T21:52:40.6012221Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20566
2023-01-11T21:52:40.6012594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6012770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6013153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6013350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6013778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6013962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6014381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6014574Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6014824Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6015070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6015475Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6015875Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6016113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6016343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6016593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.6016821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.6017221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6017618Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6017860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:52:40.6018102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:52:40.6018492Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:52:40.6018892Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:52:40.6019128Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T21:52:40.6019230Z ok (21.066s)
2023-01-11T21:52:40.6019250Z 
2023-01-11T21:52:40.6019498Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6019612Z Ran 1 test in 21.066s
2023-01-11T21:52:40.6019631Z 
2023-01-11T21:52:40.6019724Z OK
2023-01-11T21:52:40.6019744Z 
2023-01-11T21:52:40.6019867Z Generating XML reports...
2023-01-11T21:52:40.6020321Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214503.xml
2023-01-11T21:52:40.6020700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6020878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6021265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6021456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6021477Z 
2023-01-11T21:52:40.6021567Z Running tests...
2023-01-11T21:52:40.6021828Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6022143Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6022446Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6022670Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20692
2023-01-11T21:52:40.6022947Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20693
2023-01-11T21:52:40.6023332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6023556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6023922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6024116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6024483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6024660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6025034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6025228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6025476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6025726Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6026125Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6026503Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6026735Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6026965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6027207Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.6027455Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.6027855Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6028257Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6028499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:52:40.6028740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:52:40.6029120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:52:40.6029512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:52:40.6029748Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T21:52:40.6029980Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T21:52:40.6030083Z ok (21.193s)
2023-01-11T21:52:40.6030106Z 
2023-01-11T21:52:40.6030376Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6030491Z Ran 1 test in 21.193s
2023-01-11T21:52:40.6030510Z 
2023-01-11T21:52:40.6030604Z OK
2023-01-11T21:52:40.6030624Z 
2023-01-11T21:52:40.6030729Z Generating XML reports...
2023-01-11T21:52:40.6031186Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214527.xml
2023-01-11T21:52:40.6031560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6031739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6032173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6032375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6032395Z 
2023-01-11T21:52:40.6032545Z Running tests...
2023-01-11T21:52:40.6032809Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6033172Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6033440Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6033662Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20819
2023-01-11T21:52:40.6033882Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20820
2023-01-11T21:52:40.6034255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6034438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6034824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6035020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6035387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6035562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6035919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6036111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6036358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6036606Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6037013Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6037413Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6037648Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6037879Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6038020Z skip: Skipped due to small world size. (4.221s)
2023-01-11T21:52:40.6038058Z 
2023-01-11T21:52:40.6038303Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6038415Z Ran 1 test in 4.222s
2023-01-11T21:52:40.6038434Z 
2023-01-11T21:52:40.6038540Z OK (skipped=1)
2023-01-11T21:52:40.6038559Z 
2023-01-11T21:52:40.6038680Z Generating XML reports...
2023-01-11T21:52:40.6039135Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214550.xml
2023-01-11T21:52:40.6039507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6039688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6040068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6040241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6040279Z 
2023-01-11T21:52:40.6040369Z Running tests...
2023-01-11T21:52:40.6040632Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6040947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6041215Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6041487Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 20928
2023-01-11T21:52:40.6041718Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 20929
2023-01-11T21:52:40.6042137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6042294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6042675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6042866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6043231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6043408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6043789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6043980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6044399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6044660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6045048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6045451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6045681Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6045910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6046149Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2023-01-11T21:52:40.6046252Z ok (6.302s)
2023-01-11T21:52:40.6046273Z 
2023-01-11T21:52:40.6046541Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6046656Z Ran 1 test in 6.303s
2023-01-11T21:52:40.6046676Z 
2023-01-11T21:52:40.6046768Z OK
2023-01-11T21:52:40.6046788Z 
2023-01-11T21:52:40.6046894Z Generating XML reports...
2023-01-11T21:52:40.6047349Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214557.xml
2023-01-11T21:52:40.6047724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6047902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6048287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6048484Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6048504Z 
2023-01-11T21:52:40.6048616Z Running tests...
2023-01-11T21:52:40.6048879Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6049181Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6049472Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6049692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21037
2023-01-11T21:52:40.6049913Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21038
2023-01-11T21:52:40.6050286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6050464Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6050914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6051118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6051548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6051702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6052078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6052271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6052521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6052768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6053172Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6053565Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6053801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6054030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6054255Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.6054499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.6054900Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6055298Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6055523Z [E ProcessGroupGloo.cpp:138] Rank 0 timed out in monitoredBarrier after 0 ms.
2023-01-11T21:52:40.6055702Z No ranks successfully processed in monitoredBarrier.
2023-01-11T21:52:40.6055934Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2023-01-11T21:52:40.6056036Z ok (4.349s)
2023-01-11T21:52:40.6056057Z 
2023-01-11T21:52:40.6056309Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6056422Z Ran 1 test in 4.349s
2023-01-11T21:52:40.6056441Z 
2023-01-11T21:52:40.6056534Z OK
2023-01-11T21:52:40.6056553Z 
2023-01-11T21:52:40.6056680Z Generating XML reports...
2023-01-11T21:52:40.6057137Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214606.xml
2023-01-11T21:52:40.6057511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6057692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6058074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6058270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6058290Z 
2023-01-11T21:52:40.6058380Z Running tests...
2023-01-11T21:52:40.6058653Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6058970Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6059253Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6059475Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21152
2023-01-11T21:52:40.6059695Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21153
2023-01-11T21:52:40.6060116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6060299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6060667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6060922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6061296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6061473Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6061857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6062052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6062299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6062549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6062951Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6063336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6063570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6063800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6064044Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.6064291Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.6064693Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6065091Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6065327Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T21:52:40.6065429Z ok (4.421s)
2023-01-11T21:52:40.6065450Z 
2023-01-11T21:52:40.6065698Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6065811Z Ran 1 test in 4.421s
2023-01-11T21:52:40.6065830Z 
2023-01-11T21:52:40.6065924Z OK
2023-01-11T21:52:40.6065943Z 
2023-01-11T21:52:40.6066067Z Generating XML reports...
2023-01-11T21:52:40.6066522Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214613.xml
2023-01-11T21:52:40.6066895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6067074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6067456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6067634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6067671Z 
2023-01-11T21:52:40.6067762Z Running tests...
2023-01-11T21:52:40.6068025Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6068344Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6068625Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6068845Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21267
2023-01-11T21:52:40.6069064Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21268
2023-01-11T21:52:40.6069485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6069670Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6070080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6070275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6070638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6070812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6071190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6071382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6071634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6071881Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6072265Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6072669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6072900Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6073132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6073294Z skip: Skipped due to small world size. (4.151s)
2023-01-11T21:52:40.6073314Z 
2023-01-11T21:52:40.6073581Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6073693Z Ran 1 test in 4.151s
2023-01-11T21:52:40.6073713Z 
2023-01-11T21:52:40.6073820Z OK (skipped=1)
2023-01-11T21:52:40.6073842Z 
2023-01-11T21:52:40.6073967Z Generating XML reports...
2023-01-11T21:52:40.6074405Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214620.xml
2023-01-11T21:52:40.6074785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6074961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6075344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6075537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6075556Z 
2023-01-11T21:52:40.6075664Z Running tests...
2023-01-11T21:52:40.6075925Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6076244Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6076649Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2023-01-11T21:52:40.6076672Z 
2023-01-11T21:52:40.6076913Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6077026Z Ran 1 test in 0.002s
2023-01-11T21:52:40.6077045Z 
2023-01-11T21:52:40.6077153Z OK (skipped=1)
2023-01-11T21:52:40.6077172Z 
2023-01-11T21:52:40.6077295Z Generating XML reports...
2023-01-11T21:52:40.6077750Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214627.xml
2023-01-11T21:52:40.6078120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6078297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6078727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6078926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6078947Z 
2023-01-11T21:52:40.6079078Z Running tests...
2023-01-11T21:52:40.6079345Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6079658Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6080061Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2023-01-11T21:52:40.6080081Z 
2023-01-11T21:52:40.6080338Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6080452Z Ran 1 test in 0.002s
2023-01-11T21:52:40.6080472Z 
2023-01-11T21:52:40.6080578Z OK (skipped=1)
2023-01-11T21:52:40.6080597Z 
2023-01-11T21:52:40.6080718Z Generating XML reports...
2023-01-11T21:52:40.6081152Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214629.xml
2023-01-11T21:52:40.6081524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6081705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6082083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6082274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6082294Z 
2023-01-11T21:52:40.6082400Z Running tests...
2023-01-11T21:52:40.6082663Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6082976Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6083381Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2023-01-11T21:52:40.6083405Z 
2023-01-11T21:52:40.6083646Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6083759Z Ran 1 test in 0.002s
2023-01-11T21:52:40.6083781Z 
2023-01-11T21:52:40.6083890Z OK (skipped=1)
2023-01-11T21:52:40.6083910Z 
2023-01-11T21:52:40.6084030Z Generating XML reports...
2023-01-11T21:52:40.6084657Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214631.xml
2023-01-11T21:52:40.6085043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6085223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6085607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6085800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6085819Z 
2023-01-11T21:52:40.6085912Z Running tests...
2023-01-11T21:52:40.6086175Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6086488Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6086890Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2023-01-11T21:52:40.6086910Z 
2023-01-11T21:52:40.6087169Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6087279Z Ran 1 test in 0.003s
2023-01-11T21:52:40.6087298Z 
2023-01-11T21:52:40.6087401Z OK (skipped=1)
2023-01-11T21:52:40.6087420Z 
2023-01-11T21:52:40.6087543Z Generating XML reports...
2023-01-11T21:52:40.6087994Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214634.xml
2023-01-11T21:52:40.6088423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6088610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6088993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6089241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6089261Z 
2023-01-11T21:52:40.6089369Z Running tests...
2023-01-11T21:52:40.6089635Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6089949Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6090252Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2023-01-11T21:52:40.6090272Z 
2023-01-11T21:52:40.6090533Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6090627Z Ran 1 test in 0.002s
2023-01-11T21:52:40.6090650Z 
2023-01-11T21:52:40.6090758Z OK (skipped=1)
2023-01-11T21:52:40.6090777Z 
2023-01-11T21:52:40.6090901Z Generating XML reports...
2023-01-11T21:52:40.6091356Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214636.xml
2023-01-11T21:52:40.6091728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6091905Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6092284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6092474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6092493Z 
2023-01-11T21:52:40.6092584Z Running tests...
2023-01-11T21:52:40.6092846Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6093160Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6093416Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:52:40.6093438Z 
2023-01-11T21:52:40.6093699Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6093810Z Ran 1 test in 0.002s
2023-01-11T21:52:40.6093829Z 
2023-01-11T21:52:40.6093934Z OK (skipped=1)
2023-01-11T21:52:40.6093953Z 
2023-01-11T21:52:40.6094075Z Generating XML reports...
2023-01-11T21:52:40.6094522Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214639.xml
2023-01-11T21:52:40.6094874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6095052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6095438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6095633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6095652Z 
2023-01-11T21:52:40.6095763Z Running tests...
2023-01-11T21:52:40.6096026Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6096340Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6096611Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:52:40.6096630Z 
2023-01-11T21:52:40.6096888Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6096980Z Ran 1 test in 0.002s
2023-01-11T21:52:40.6096999Z 
2023-01-11T21:52:40.6097110Z OK (skipped=1)
2023-01-11T21:52:40.6097129Z 
2023-01-11T21:52:40.6097252Z Generating XML reports...
2023-01-11T21:52:40.6097749Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214641.xml
2023-01-11T21:52:40.6098130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6098355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6098738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6098930Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6098949Z 
2023-01-11T21:52:40.6099057Z Running tests...
2023-01-11T21:52:40.6099302Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6099618Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6099931Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:52:40.6099956Z 
2023-01-11T21:52:40.6100215Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6100325Z Ran 1 test in 0.002s
2023-01-11T21:52:40.6100347Z 
2023-01-11T21:52:40.6100452Z OK (skipped=1)
2023-01-11T21:52:40.6100471Z 
2023-01-11T21:52:40.6100593Z Generating XML reports...
2023-01-11T21:52:40.6101041Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214643.xml
2023-01-11T21:52:40.6101389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6101566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6101946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6102137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6102156Z 
2023-01-11T21:52:40.6102270Z Running tests...
2023-01-11T21:52:40.6102532Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6102847Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6103158Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6103379Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21640
2023-01-11T21:52:40.6103578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21641
2023-01-11T21:52:40.6103953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6104130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6104515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6104709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6105076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6105254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6105630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6105802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6106050Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6106296Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6106698Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6107143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6107383Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6107656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6107762Z ok (4.238s)
2023-01-11T21:52:40.6107783Z 
2023-01-11T21:52:40.6108048Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6108141Z Ran 1 test in 4.238s
2023-01-11T21:52:40.6108161Z 
2023-01-11T21:52:40.6108253Z OK
2023-01-11T21:52:40.6108272Z 
2023-01-11T21:52:40.6108395Z Generating XML reports...
2023-01-11T21:52:40.6108853Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214646.xml
2023-01-11T21:52:40.6109229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6109405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6109786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6109981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6110001Z 
2023-01-11T21:52:40.6110108Z Running tests...
2023-01-11T21:52:40.6110349Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6110664Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6110960Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6111182Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21749
2023-01-11T21:52:40.6111404Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21750
2023-01-11T21:52:40.6111775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6111955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6112335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6112508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6112871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6113044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6113420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6113610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6113859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6114105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6114509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6114899Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6115114Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6115341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6115444Z ok (4.149s)
2023-01-11T21:52:40.6115463Z 
2023-01-11T21:52:40.6115730Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6115841Z Ran 1 test in 4.149s
2023-01-11T21:52:40.6115861Z 
2023-01-11T21:52:40.6116001Z OK
2023-01-11T21:52:40.6116023Z 
2023-01-11T21:52:40.6116150Z Generating XML reports...
2023-01-11T21:52:40.6116608Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214653.xml
2023-01-11T21:52:40.6117028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6117187Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6117566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6117756Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6117775Z 
2023-01-11T21:52:40.6117882Z Running tests...
2023-01-11T21:52:40.6118146Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6118463Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6118747Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:52:40.6118770Z 
2023-01-11T21:52:40.6119031Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6119124Z Ran 1 test in 0.002s
2023-01-11T21:52:40.6119160Z 
2023-01-11T21:52:40.6119249Z OK (skipped=1)
2023-01-11T21:52:40.6119268Z 
2023-01-11T21:52:40.6119391Z Generating XML reports...
2023-01-11T21:52:40.6119842Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214659.xml
2023-01-11T21:52:40.6120217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6120393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6120776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6120966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6120985Z 
2023-01-11T21:52:40.6121094Z Running tests...
2023-01-11T21:52:40.6121339Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6121653Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6121954Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T21:52:40.6121974Z 
2023-01-11T21:52:40.6122234Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6122343Z Ran 1 test in 0.002s
2023-01-11T21:52:40.6122362Z 
2023-01-11T21:52:40.6122469Z OK (skipped=1)
2023-01-11T21:52:40.6122488Z 
2023-01-11T21:52:40.6122609Z Generating XML reports...
2023-01-11T21:52:40.6123062Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214702.xml
2023-01-11T21:52:40.6123435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6123598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6123980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6124170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6124345Z 
2023-01-11T21:52:40.6124471Z Running tests...
2023-01-11T21:52:40.6124737Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6125054Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6125333Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6126196Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.615s)
2023-01-11T21:52:40.6126273Z 
2023-01-11T21:52:40.6126546Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6126641Z Ran 1 test in 1.615s
2023-01-11T21:52:40.6126675Z 
2023-01-11T21:52:40.6126764Z OK (skipped=1)
2023-01-11T21:52:40.6126783Z 
2023-01-11T21:52:40.6126906Z Generating XML reports...
2023-01-11T21:52:40.6127359Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214704.xml
2023-01-11T21:52:40.6127731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6127914Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6128463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6128668Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6128687Z 
2023-01-11T21:52:40.6128797Z Running tests...
2023-01-11T21:52:40.6129046Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6129364Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6129648Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6129873Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 21958
2023-01-11T21:52:40.6130089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 21959
2023-01-11T21:52:40.6130468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6130643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6131028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6131220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6131569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6131743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6132119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6132408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6132665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6132917Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6133359Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6133763Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6133980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6134214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6134316Z ok (5.550s)
2023-01-11T21:52:40.6134336Z 
2023-01-11T21:52:40.6134601Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6134711Z Ran 1 test in 5.550s
2023-01-11T21:52:40.6134731Z 
2023-01-11T21:52:40.6134825Z OK
2023-01-11T21:52:40.6134845Z 
2023-01-11T21:52:40.6135024Z Generating XML reports...
2023-01-11T21:52:40.6135487Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214708.xml
2023-01-11T21:52:40.6135910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6136069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6136449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6136641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6136661Z 
2023-01-11T21:52:40.6136769Z Running tests...
2023-01-11T21:52:40.6137028Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6137343Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6137616Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6137840Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22073
2023-01-11T21:52:40.6138045Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22074
2023-01-11T21:52:40.6138412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6138586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6138969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6139159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6139524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6139699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6140077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6140269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6140499Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6140746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6141147Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6141546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6141780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6142013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6142115Z ok (5.625s)
2023-01-11T21:52:40.6142135Z 
2023-01-11T21:52:40.6142401Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6142497Z Ran 1 test in 5.625s
2023-01-11T21:52:40.6142533Z 
2023-01-11T21:52:40.6142608Z OK
2023-01-11T21:52:40.6142628Z 
2023-01-11T21:52:40.6142748Z Generating XML reports...
2023-01-11T21:52:40.6143202Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214716.xml
2023-01-11T21:52:40.6143573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6143743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6144122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6144366Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6144388Z 
2023-01-11T21:52:40.6144501Z Running tests...
2023-01-11T21:52:40.6144750Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6145114Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6145403Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6145623Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22185
2023-01-11T21:52:40.6145841Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22186
2023-01-11T21:52:40.6146214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6146386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6146772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6146963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6147313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6147491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6147866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6148054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6148303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6148549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6148957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6149354Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6149569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6149794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6149899Z ok (5.648s)
2023-01-11T21:52:40.6149918Z 
2023-01-11T21:52:40.6150179Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6150289Z Ran 1 test in 5.648s
2023-01-11T21:52:40.6150309Z 
2023-01-11T21:52:40.6150400Z OK
2023-01-11T21:52:40.6150419Z 
2023-01-11T21:52:40.6150541Z Generating XML reports...
2023-01-11T21:52:40.6150995Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214724.xml
2023-01-11T21:52:40.6151369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6151529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6151917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6152105Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6152125Z 
2023-01-11T21:52:40.6152230Z Running tests...
2023-01-11T21:52:40.6152490Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6152806Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6153086Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6153886Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.625s)
2023-01-11T21:52:40.6153943Z 
2023-01-11T21:52:40.6154216Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6154311Z Ran 1 test in 1.625s
2023-01-11T21:52:40.6154347Z 
2023-01-11T21:52:40.6154437Z OK (skipped=1)
2023-01-11T21:52:40.6154455Z 
2023-01-11T21:52:40.6154580Z Generating XML reports...
2023-01-11T21:52:40.6155026Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214733.xml
2023-01-11T21:52:40.6155401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6155577Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6155962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6156155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6156175Z 
2023-01-11T21:52:40.6156280Z Running tests...
2023-01-11T21:52:40.6156525Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6156840Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6157136Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6157886Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.627s)
2023-01-11T21:52:40.6157906Z 
2023-01-11T21:52:40.6158165Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6158278Z Ran 1 test in 1.627s
2023-01-11T21:52:40.6158298Z 
2023-01-11T21:52:40.6158403Z OK (skipped=1)
2023-01-11T21:52:40.6158422Z 
2023-01-11T21:52:40.6158547Z Generating XML reports...
2023-01-11T21:52:40.6159000Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214737.xml
2023-01-11T21:52:40.6159373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6159531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6159911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6160101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6160120Z 
2023-01-11T21:52:40.6160229Z Running tests...
2023-01-11T21:52:40.6160497Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6160815Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6161128Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6161354Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22365
2023-01-11T21:52:40.6161555Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22366
2023-01-11T21:52:40.6161930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6162108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6162486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6162679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6163098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6163276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6163695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6163886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6164116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6164543Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6164962Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6165366Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6165598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6165826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6165978Z skip: Need at least 4 CUDA devices (4.112s)
2023-01-11T21:52:40.6165998Z 
2023-01-11T21:52:40.6166261Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6166372Z Ran 1 test in 4.112s
2023-01-11T21:52:40.6166391Z 
2023-01-11T21:52:40.6166480Z OK (skipped=1)
2023-01-11T21:52:40.6166499Z 
2023-01-11T21:52:40.6166625Z Generating XML reports...
2023-01-11T21:52:40.6167078Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214741.xml
2023-01-11T21:52:40.6167450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6167632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6168009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6168207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6168228Z 
2023-01-11T21:52:40.6168335Z Running tests...
2023-01-11T21:52:40.6168582Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6168899Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6169226Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6169447Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22474
2023-01-11T21:52:40.6169668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22475
2023-01-11T21:52:40.6170045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6170220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6170605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6170798Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6171143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6171319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6171689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6171878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6172204Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6172462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6172930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6173326Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6173560Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6173772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6173924Z skip: Need at least 4 CUDA devices (4.145s)
2023-01-11T21:52:40.6173944Z 
2023-01-11T21:52:40.6174207Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6174315Z Ran 1 test in 4.145s
2023-01-11T21:52:40.6174336Z 
2023-01-11T21:52:40.6174446Z OK (skipped=1)
2023-01-11T21:52:40.6174466Z 
2023-01-11T21:52:40.6174587Z Generating XML reports...
2023-01-11T21:52:40.6175042Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214748.xml
2023-01-11T21:52:40.6175417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6175575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6175956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6176149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6176169Z 
2023-01-11T21:52:40.6176277Z Running tests...
2023-01-11T21:52:40.6176540Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6176857Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6177144Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6177901Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.626s)
2023-01-11T21:52:40.6177922Z 
2023-01-11T21:52:40.6178191Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6178301Z Ran 1 test in 1.626s
2023-01-11T21:52:40.6178321Z 
2023-01-11T21:52:40.6178409Z OK (skipped=1)
2023-01-11T21:52:40.6178428Z 
2023-01-11T21:52:40.6178553Z Generating XML reports...
2023-01-11T21:52:40.6179005Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214754.xml
2023-01-11T21:52:40.6179377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6179554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6179938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6180131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6180150Z 
2023-01-11T21:52:40.6180256Z Running tests...
2023-01-11T21:52:40.6180518Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6180818Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6181085Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6181306Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22617
2023-01-11T21:52:40.6181575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22618
2023-01-11T21:52:40.6181957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6182178Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6182558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6182750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6183097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6183273Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6183641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6183835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6184084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6184489Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6184733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6185131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6185359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6185571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6185816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.6186060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.6186460Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6186860Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6187199Z STAGE:2023-01-11 21:48:02 22618:22618 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6187528Z STAGE:2023-01-11 21:48:02 22617:22617 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6188072Z STAGE:2023-01-11 21:48:02 22618:22618 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:48:02 22617:22617 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6188092Z 
2023-01-11T21:52:40.6188668Z STAGE:2023-01-11 21:48:02 22618:22618 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:48:02 22617:22617 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6188688Z 
2023-01-11T21:52:40.6189018Z STAGE:2023-01-11 21:48:02 22618:22618 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6189346Z STAGE:2023-01-11 21:48:02 22617:22617 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6189664Z STAGE:2023-01-11 21:48:02 22617:22617 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6190000Z STAGE:2023-01-11 21:48:02 22618:22618 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6190344Z STAGE:2023-01-11 21:48:02 22617:22617 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6190693Z STAGE:2023-01-11 21:48:02 22618:22618 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6190794Z ok (4.249s)
2023-01-11T21:52:40.6190814Z 
2023-01-11T21:52:40.6191124Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6191240Z Ran 1 test in 4.249s
2023-01-11T21:52:40.6191260Z 
2023-01-11T21:52:40.6191403Z OK
2023-01-11T21:52:40.6191423Z 
2023-01-11T21:52:40.6191530Z Generating XML reports...
2023-01-11T21:52:40.6191989Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214758.xml
2023-01-11T21:52:40.6192364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6192545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6192927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6193116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6193135Z 
2023-01-11T21:52:40.6193239Z Running tests...
2023-01-11T21:52:40.6193505Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6193822Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6194074Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6194295Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22736
2023-01-11T21:52:40.6194513Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22737
2023-01-11T21:52:40.6194887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6195062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6195446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6195642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6196006Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6196164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6196535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6196720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6196970Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6197215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6197615Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6198016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6198248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6198477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6198701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.6198943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.6199342Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6199736Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6200075Z STAGE:2023-01-11 21:48:09 22737:22737 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6200450Z STAGE:2023-01-11 21:48:09 22736:22736 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6200799Z STAGE:2023-01-11 21:48:09 22737:22737 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6201174Z STAGE:2023-01-11 21:48:09 22736:22736 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6201521Z STAGE:2023-01-11 21:48:09 22737:22737 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6201852Z STAGE:2023-01-11 21:48:09 22736:22736 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6202179Z STAGE:2023-01-11 21:48:09 22737:22737 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6202497Z STAGE:2023-01-11 21:48:09 22736:22736 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6203054Z STAGE:2023-01-11 21:48:09 22736:22736 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:48:09 22737:22737 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6203075Z 
2023-01-11T21:52:40.6203653Z STAGE:2023-01-11 21:48:09 22737:22737 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:48:09 22736:22736 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6203677Z 
2023-01-11T21:52:40.6203778Z ok (4.230s)
2023-01-11T21:52:40.6203797Z 
2023-01-11T21:52:40.6204061Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6204173Z Ran 1 test in 4.230s
2023-01-11T21:52:40.6204425Z 
2023-01-11T21:52:40.6204530Z OK
2023-01-11T21:52:40.6204549Z 
2023-01-11T21:52:40.6204677Z Generating XML reports...
2023-01-11T21:52:40.6205126Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214805.xml
2023-01-11T21:52:40.6205503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6205684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6206065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6206263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6206282Z 
2023-01-11T21:52:40.6206391Z Running tests...
2023-01-11T21:52:40.6206655Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6206971Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6207242Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6207446Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22855
2023-01-11T21:52:40.6207661Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22856
2023-01-11T21:52:40.6208038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6208215Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6208603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6208794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6209166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6209341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6209695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6209885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6210206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6210456Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6210913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6211315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6211548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6211782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6212026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.6212251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.6212655Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6213047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6213388Z STAGE:2023-01-11 21:48:16 22856:22856 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6213715Z STAGE:2023-01-11 21:48:16 22855:22855 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6214261Z STAGE:2023-01-11 21:48:16 22855:22855 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:48:16 22856:22856 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6214282Z 
2023-01-11T21:52:40.6214864Z STAGE:2023-01-11 21:48:16 22855:22855 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:48:16 22856:22856 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6214885Z 
2023-01-11T21:52:40.6215214Z STAGE:2023-01-11 21:48:16 22856:22856 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6215539Z STAGE:2023-01-11 21:48:16 22855:22855 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6215871Z STAGE:2023-01-11 21:48:16 22856:22856 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6216198Z STAGE:2023-01-11 21:48:16 22855:22855 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6216526Z STAGE:2023-01-11 21:48:16 22856:22856 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6216875Z STAGE:2023-01-11 21:48:16 22855:22855 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6216977Z ok (4.246s)
2023-01-11T21:52:40.6216996Z 
2023-01-11T21:52:40.6217261Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6217375Z Ran 1 test in 4.246s
2023-01-11T21:52:40.6217395Z 
2023-01-11T21:52:40.6217486Z OK
2023-01-11T21:52:40.6217505Z 
2023-01-11T21:52:40.6217627Z Generating XML reports...
2023-01-11T21:52:40.6218084Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214812.xml
2023-01-11T21:52:40.6218457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6218617Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6219000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6219192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6219213Z 
2023-01-11T21:52:40.6219321Z Running tests...
2023-01-11T21:52:40.6219583Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6219949Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6220223Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6220483Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 22974
2023-01-11T21:52:40.6220682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 22975
2023-01-11T21:52:40.6221059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6221235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6221613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6221808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6222174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6222349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6222723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6222911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6223140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6223384Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6223785Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6224186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6224421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6224653Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6224896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.6225138Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.6225536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6225913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6226249Z STAGE:2023-01-11 21:48:23 22974:22974 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6226576Z STAGE:2023-01-11 21:48:23 22975:22975 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6227124Z STAGE:2023-01-11 21:48:23 22975:22975 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:48:23 22974:22974 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6227148Z 
2023-01-11T21:52:40.6227721Z STAGE:2023-01-11 21:48:23 22974:22974 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:48:23 22975:22975 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6227742Z 
2023-01-11T21:52:40.6228070Z STAGE:2023-01-11 21:48:23 22975:22975 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6228394Z STAGE:2023-01-11 21:48:23 22974:22974 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6228731Z STAGE:2023-01-11 21:48:23 22975:22975 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6229115Z STAGE:2023-01-11 21:48:23 22974:22974 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6229475Z STAGE:2023-01-11 21:48:23 22975:22975 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6229848Z STAGE:2023-01-11 21:48:23 22974:22974 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6229953Z ok (4.236s)
2023-01-11T21:52:40.6229972Z 
2023-01-11T21:52:40.6230232Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6230344Z Ran 1 test in 4.236s
2023-01-11T21:52:40.6230364Z 
2023-01-11T21:52:40.6230455Z OK
2023-01-11T21:52:40.6230475Z 
2023-01-11T21:52:40.6230601Z Generating XML reports...
2023-01-11T21:52:40.6231054Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214819.xml
2023-01-11T21:52:40.6231419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6231595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6231960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6232156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6232176Z 
2023-01-11T21:52:40.6232283Z Running tests...
2023-01-11T21:52:40.6232547Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6232863Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6233169Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6233397Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23093
2023-01-11T21:52:40.6233617Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23094
2023-01-11T21:52:40.6233975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6234147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6234527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6234723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6235085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6235262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6235637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6235824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6236070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6236300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6236703Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6237100Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6237332Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6237559Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6237721Z skip: Skipped due to small world size. (4.238s)
2023-01-11T21:52:40.6237741Z 
2023-01-11T21:52:40.6238005Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6238112Z Ran 1 test in 4.238s
2023-01-11T21:52:40.6238132Z 
2023-01-11T21:52:40.6238239Z OK (skipped=1)
2023-01-11T21:52:40.6238258Z 
2023-01-11T21:52:40.6238415Z Generating XML reports...
2023-01-11T21:52:40.6238880Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214826.xml
2023-01-11T21:52:40.6239302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6239476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6239861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6240055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6240075Z 
2023-01-11T21:52:40.6240182Z Running tests...
2023-01-11T21:52:40.6240443Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6240741Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6240998Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6241211Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23202
2023-01-11T21:52:40.6241431Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23203
2023-01-11T21:52:40.6241813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6241990Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6242371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6242557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6242920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6243095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6243473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6243648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6243893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6244136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6244776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6245176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6245406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6245638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6245795Z skip: Skipped due to small world size. (4.134s)
2023-01-11T21:52:40.6245815Z 
2023-01-11T21:52:40.6246086Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6246180Z Ran 1 test in 4.134s
2023-01-11T21:52:40.6246200Z 
2023-01-11T21:52:40.6246304Z OK (skipped=1)
2023-01-11T21:52:40.6246323Z 
2023-01-11T21:52:40.6246442Z Generating XML reports...
2023-01-11T21:52:40.6246893Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214832.xml
2023-01-11T21:52:40.6247261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6247435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6247818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6248083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6248105Z 
2023-01-11T21:52:40.6248201Z Running tests...
2023-01-11T21:52:40.6248516Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6248828Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6249095Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6249312Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23311
2023-01-11T21:52:40.6249527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23312
2023-01-11T21:52:40.6249896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6250068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6250433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6250625Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6250989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6251157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6251529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6251715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6251960Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6252202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6252608Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6252986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6253220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6253446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6253603Z skip: Skipped due to small world size. (4.143s)
2023-01-11T21:52:40.6253622Z 
2023-01-11T21:52:40.6253885Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6253993Z Ran 1 test in 4.144s
2023-01-11T21:52:40.6254012Z 
2023-01-11T21:52:40.6254114Z OK (skipped=1)
2023-01-11T21:52:40.6254133Z 
2023-01-11T21:52:40.6254253Z Generating XML reports...
2023-01-11T21:52:40.6254705Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214839.xml
2023-01-11T21:52:40.6255062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6255238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6255614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6255804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6255823Z 
2023-01-11T21:52:40.6255922Z Running tests...
2023-01-11T21:52:40.6256186Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6256499Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6256755Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6257007Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23420
2023-01-11T21:52:40.6257229Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23421
2023-01-11T21:52:40.6257603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6257833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6258213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6258404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6258762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6258932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6259301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6259475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6259720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6259966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6260365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6260759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6260988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6261219Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6261375Z skip: Skipped due to small world size. (4.229s)
2023-01-11T21:52:40.6261394Z 
2023-01-11T21:52:40.6261655Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6261749Z Ran 1 test in 4.229s
2023-01-11T21:52:40.6261768Z 
2023-01-11T21:52:40.6261871Z OK (skipped=1)
2023-01-11T21:52:40.6261894Z 
2023-01-11T21:52:40.6262012Z Generating XML reports...
2023-01-11T21:52:40.6262462Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214846.xml
2023-01-11T21:52:40.6262831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6263004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6263378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6263567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6263586Z 
2023-01-11T21:52:40.6263690Z Running tests...
2023-01-11T21:52:40.6263938Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6264246Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6264496Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6264712Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23529
2023-01-11T21:52:40.6264927Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23530
2023-01-11T21:52:40.6265295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6265469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6265847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6266020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6266431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6266610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6267078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6267262Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6267503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6267745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6268143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6268543Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6268760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6268986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6269324Z STAGE:2023-01-11 21:48:56 23529:23529 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6269647Z STAGE:2023-01-11 21:48:56 23530:23530 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6269983Z STAGE:2023-01-11 21:48:56 23530:23530 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6270314Z STAGE:2023-01-11 21:48:56 23529:23529 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6270659Z STAGE:2023-01-11 21:48:56 23530:23530 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6271007Z STAGE:2023-01-11 21:48:56 23529:23529 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6271321Z STAGE:2023-01-11 21:48:56 23530:23530 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6271652Z STAGE:2023-01-11 21:48:56 23529:23529 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6271982Z STAGE:2023-01-11 21:48:56 23529:23529 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6272303Z STAGE:2023-01-11 21:48:56 23530:23530 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6272646Z STAGE:2023-01-11 21:48:56 23529:23529 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6272987Z STAGE:2023-01-11 21:48:56 23530:23530 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6273086Z ok (4.202s)
2023-01-11T21:52:40.6273106Z 
2023-01-11T21:52:40.6273364Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6273477Z Ran 1 test in 4.202s
2023-01-11T21:52:40.6273497Z 
2023-01-11T21:52:40.6273571Z OK
2023-01-11T21:52:40.6273589Z 
2023-01-11T21:52:40.6273710Z Generating XML reports...
2023-01-11T21:52:40.6274169Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214852.xml
2023-01-11T21:52:40.6274538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6274712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6275093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6275284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6275303Z 
2023-01-11T21:52:40.6275409Z Running tests...
2023-01-11T21:52:40.6275668Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6276015Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6276269Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6276528Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23642
2023-01-11T21:52:40.6276744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23643
2023-01-11T21:52:40.6277115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6277289Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6277674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6277865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6278223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6278395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6278772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6278965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6279209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6279451Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6279849Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6280244Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6280474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6280687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6281024Z STAGE:2023-01-11 21:49:03 23643:23643 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6281346Z STAGE:2023-01-11 21:49:03 23642:23642 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6281679Z STAGE:2023-01-11 21:49:03 23643:23643 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6282013Z STAGE:2023-01-11 21:49:03 23642:23642 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6282586Z STAGE:2023-01-11 21:49:03 23642:23642 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:49:03 23643:23643 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6282607Z 
2023-01-11T21:52:40.6282941Z STAGE:2023-01-11 21:49:03 23643:23643 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6283263Z STAGE:2023-01-11 21:49:03 23642:23642 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6283598Z STAGE:2023-01-11 21:49:03 23642:23642 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6283921Z STAGE:2023-01-11 21:49:03 23643:23643 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6284464Z STAGE:2023-01-11 21:49:03 23642:23642 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6284827Z STAGE:2023-01-11 21:49:03 23643:23643 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6284925Z ok (4.326s)
2023-01-11T21:52:40.6284945Z 
2023-01-11T21:52:40.6285208Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6285317Z Ran 1 test in 4.326s
2023-01-11T21:52:40.6285337Z 
2023-01-11T21:52:40.6285428Z OK
2023-01-11T21:52:40.6285522Z 
2023-01-11T21:52:40.6285653Z Generating XML reports...
2023-01-11T21:52:40.6286113Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214859.xml
2023-01-11T21:52:40.6286531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6286709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6287093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6287281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6287300Z 
2023-01-11T21:52:40.6287405Z Running tests...
2023-01-11T21:52:40.6287660Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6287974Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6288256Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2023-01-11T21:52:40.6288276Z 
2023-01-11T21:52:40.6288540Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6288634Z Ran 1 test in 0.002s
2023-01-11T21:52:40.6288653Z 
2023-01-11T21:52:40.6288759Z OK (skipped=1)
2023-01-11T21:52:40.6288778Z 
2023-01-11T21:52:40.6288899Z Generating XML reports...
2023-01-11T21:52:40.6289348Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214906.xml
2023-01-11T21:52:40.6289720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6289894Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6290273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6290466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6290486Z 
2023-01-11T21:52:40.6290589Z Running tests...
2023-01-11T21:52:40.6290836Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6291194Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6291454Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6291672Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23788
2023-01-11T21:52:40.6291890Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23789
2023-01-11T21:52:40.6292258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6292431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6292811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6292988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6293354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6293525Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6293900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6294088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6294333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6294578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6295030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6295438Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6295706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6295932Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6296266Z STAGE:2023-01-11 21:49:12 23788:23788 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6296585Z STAGE:2023-01-11 21:49:12 23789:23789 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6297131Z STAGE:2023-01-11 21:49:12 23788:23788 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:49:12 23789:23789 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6297152Z 
2023-01-11T21:52:40.6297718Z STAGE:2023-01-11 21:49:12 23788:23788 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:49:12 23789:23789 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6297741Z 
2023-01-11T21:52:40.6298069Z STAGE:2023-01-11 21:49:12 23789:23789 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6298387Z STAGE:2023-01-11 21:49:12 23788:23788 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6298719Z STAGE:2023-01-11 21:49:12 23788:23788 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6299051Z STAGE:2023-01-11 21:49:12 23789:23789 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6299380Z STAGE:2023-01-11 21:49:12 23788:23788 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6299726Z STAGE:2023-01-11 21:49:12 23789:23789 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6299828Z ok (4.256s)
2023-01-11T21:52:40.6299848Z 
2023-01-11T21:52:40.6300109Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6300219Z Ran 1 test in 4.257s
2023-01-11T21:52:40.6300238Z 
2023-01-11T21:52:40.6300330Z OK
2023-01-11T21:52:40.6300349Z 
2023-01-11T21:52:40.6300471Z Generating XML reports...
2023-01-11T21:52:40.6300924Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214908.xml
2023-01-11T21:52:40.6301290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6301450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6301829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6302017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6302040Z 
2023-01-11T21:52:40.6302146Z Running tests...
2023-01-11T21:52:40.6302404Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6302718Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6303012Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s)
2023-01-11T21:52:40.6303033Z 
2023-01-11T21:52:40.6303292Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6303384Z Ran 1 test in 0.002s
2023-01-11T21:52:40.6303419Z 
2023-01-11T21:52:40.6303508Z OK (skipped=1)
2023-01-11T21:52:40.6303527Z 
2023-01-11T21:52:40.6303647Z Generating XML reports...
2023-01-11T21:52:40.6304092Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214915.xml
2023-01-11T21:52:40.6304517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6304699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6305125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6305315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6305335Z 
2023-01-11T21:52:40.6305438Z Running tests...
2023-01-11T21:52:40.6305683Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6305994Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6306267Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s)
2023-01-11T21:52:40.6306288Z 
2023-01-11T21:52:40.6306548Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6306661Z Ran 1 test in 0.003s
2023-01-11T21:52:40.6306680Z 
2023-01-11T21:52:40.6306785Z OK (skipped=1)
2023-01-11T21:52:40.6306804Z 
2023-01-11T21:52:40.6306926Z Generating XML reports...
2023-01-11T21:52:40.6307383Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214918.xml
2023-01-11T21:52:40.6307753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6307911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6308290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6308477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6308497Z 
2023-01-11T21:52:40.6308604Z Running tests...
2023-01-11T21:52:40.6308864Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6309180Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6309426Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6309648Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 23967
2023-01-11T21:52:40.6309847Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 23968
2023-01-11T21:52:40.6310216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6310387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6310761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6310949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6311311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6311483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6311859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6312048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6312277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6312517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6312917Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6313316Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6313596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6313830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6314242Z STAGE:2023-01-11 21:49:24 23967:23967 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6314565Z STAGE:2023-01-11 21:49:24 23968:23968 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6314900Z STAGE:2023-01-11 21:49:24 23968:23968 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6315211Z STAGE:2023-01-11 21:49:24 23967:23967 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6315556Z STAGE:2023-01-11 21:49:24 23968:23968 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6315901Z STAGE:2023-01-11 21:49:24 23967:23967 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6316228Z STAGE:2023-01-11 21:49:24 23968:23968 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6316549Z STAGE:2023-01-11 21:49:24 23967:23967 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6316881Z STAGE:2023-01-11 21:49:24 23967:23967 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6317205Z STAGE:2023-01-11 21:49:24 23968:23968 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6317547Z STAGE:2023-01-11 21:49:24 23967:23967 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6317889Z STAGE:2023-01-11 21:49:24 23968:23968 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6317974Z ok (4.238s)
2023-01-11T21:52:40.6317994Z 
2023-01-11T21:52:40.6318256Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6318367Z Ran 1 test in 4.238s
2023-01-11T21:52:40.6318387Z 
2023-01-11T21:52:40.6318477Z OK
2023-01-11T21:52:40.6318496Z 
2023-01-11T21:52:40.6318621Z Generating XML reports...
2023-01-11T21:52:40.6319073Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214920.xml
2023-01-11T21:52:40.6319450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6319624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6319991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6320179Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6320199Z 
2023-01-11T21:52:40.6320302Z Running tests...
2023-01-11T21:52:40.6320559Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6320873Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6321132Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2023-01-11T21:52:40.6321152Z 
2023-01-11T21:52:40.6321410Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6321520Z Ran 1 test in 0.002s
2023-01-11T21:52:40.6321540Z 
2023-01-11T21:52:40.6321642Z OK (skipped=1)
2023-01-11T21:52:40.6321660Z 
2023-01-11T21:52:40.6321765Z Generating XML reports...
2023-01-11T21:52:40.6322214Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214927.xml
2023-01-11T21:52:40.6322585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6322758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6323137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6323378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6323398Z 
2023-01-11T21:52:40.6323508Z Running tests...
2023-01-11T21:52:40.6323775Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6324148Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6324622Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2023-01-11T21:52:40.6324644Z 
2023-01-11T21:52:40.6324912Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6325020Z Ran 1 test in 0.002s
2023-01-11T21:52:40.6325039Z 
2023-01-11T21:52:40.6325144Z OK (skipped=1)
2023-01-11T21:52:40.6325164Z 
2023-01-11T21:52:40.6325285Z Generating XML reports...
2023-01-11T21:52:40.6325734Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214929.xml
2023-01-11T21:52:40.6326107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6326282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6326652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6326841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6326860Z 
2023-01-11T21:52:40.6326962Z Running tests...
2023-01-11T21:52:40.6327223Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6327534Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6327790Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6328008Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24146
2023-01-11T21:52:40.6328227Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24147
2023-01-11T21:52:40.6328596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6328759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6329137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6329323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6329685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6329857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6330227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6330418Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6330664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6330893Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6331291Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6331687Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6331916Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6332142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6332475Z STAGE:2023-01-11 21:49:36 24147:24147 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6332873Z STAGE:2023-01-11 21:49:36 24146:24146 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6333265Z STAGE:2023-01-11 21:49:36 24147:24147 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6333661Z STAGE:2023-01-11 21:49:36 24146:24146 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6333992Z STAGE:2023-01-11 21:49:36 24147:24147 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6334334Z STAGE:2023-01-11 21:49:36 24146:24146 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6334661Z STAGE:2023-01-11 21:49:36 24147:24147 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6334981Z STAGE:2023-01-11 21:49:36 24146:24146 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6335314Z STAGE:2023-01-11 21:49:36 24146:24146 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6335643Z STAGE:2023-01-11 21:49:36 24147:24147 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6335984Z STAGE:2023-01-11 21:49:36 24146:24146 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6336326Z STAGE:2023-01-11 21:49:36 24147:24147 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6336428Z ok (4.208s)
2023-01-11T21:52:40.6336448Z 
2023-01-11T21:52:40.6336693Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6336801Z Ran 1 test in 4.208s
2023-01-11T21:52:40.6336820Z 
2023-01-11T21:52:40.6336909Z OK
2023-01-11T21:52:40.6336928Z 
2023-01-11T21:52:40.6337049Z Generating XML reports...
2023-01-11T21:52:40.6337499Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214932.xml
2023-01-11T21:52:40.6337870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6338045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6338426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6338618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6338637Z 
2023-01-11T21:52:40.6338727Z Running tests...
2023-01-11T21:52:40.6338986Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6339301Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6339542Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6339759Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24259
2023-01-11T21:52:40.6339974Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24260
2023-01-11T21:52:40.6340345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6340519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6340885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6341074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6341437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6341611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6341984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6342173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6342469Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6342720Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6343165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6343546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6343774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6344000Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6344333Z STAGE:2023-01-11 21:49:42 24260:24260 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6344654Z STAGE:2023-01-11 21:49:42 24259:24259 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6344986Z STAGE:2023-01-11 21:49:42 24259:24259 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6345317Z STAGE:2023-01-11 21:49:42 24260:24260 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6345665Z STAGE:2023-01-11 21:49:42 24259:24259 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6346011Z STAGE:2023-01-11 21:49:42 24260:24260 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6346321Z STAGE:2023-01-11 21:49:42 24259:24259 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6346638Z STAGE:2023-01-11 21:49:42 24260:24260 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6346972Z STAGE:2023-01-11 21:49:42 24260:24260 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6347536Z STAGE:2023-01-11 21:49:42 24260:24260 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:49:42 24259:24259 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6347556Z 
2023-01-11T21:52:40.6347897Z STAGE:2023-01-11 21:49:42 24259:24259 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6348000Z ok (4.143s)
2023-01-11T21:52:40.6348020Z 
2023-01-11T21:52:40.6348281Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6348388Z Ran 1 test in 4.143s
2023-01-11T21:52:40.6348408Z 
2023-01-11T21:52:40.6348494Z OK
2023-01-11T21:52:40.6348512Z 
2023-01-11T21:52:40.6348618Z Generating XML reports...
2023-01-11T21:52:40.6349072Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214938.xml
2023-01-11T21:52:40.6349447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6349623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6350000Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6350195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6350215Z 
2023-01-11T21:52:40.6350321Z Running tests...
2023-01-11T21:52:40.6350579Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6350890Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6351130Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6351349Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24372
2023-01-11T21:52:40.6351558Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24373
2023-01-11T21:52:40.6351971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6352151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6352530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6352763Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6353126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6353281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6353653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6353834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6354081Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6354328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6354726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6355125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6355356Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6355585Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6355669Z ok (4.246s)
2023-01-11T21:52:40.6355688Z 
2023-01-11T21:52:40.6355946Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6356055Z Ran 1 test in 4.247s
2023-01-11T21:52:40.6356074Z 
2023-01-11T21:52:40.6356165Z OK
2023-01-11T21:52:40.6356184Z 
2023-01-11T21:52:40.6356305Z Generating XML reports...
2023-01-11T21:52:40.6356755Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214945.xml
2023-01-11T21:52:40.6357127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6357305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6357666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6357855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6357874Z 
2023-01-11T21:52:40.6357979Z Running tests...
2023-01-11T21:52:40.6358235Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6358546Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6358805Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6359023Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24481
2023-01-11T21:52:40.6359239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24482
2023-01-11T21:52:40.6359612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6359769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6360147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6360334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6360695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6360866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6361293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6361489Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6361776Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6362003Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6362407Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6362804Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6363033Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6363261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6363602Z STAGE:2023-01-11 21:49:56 24481:24481 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6363924Z STAGE:2023-01-11 21:49:56 24482:24482 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6364699Z STAGE:2023-01-11 21:49:56 24481:24481 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:49:56 24482:24482 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6364722Z 
2023-01-11T21:52:40.6365302Z STAGE:2023-01-11 21:49:56 24482:24482 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:49:56 24481:24481 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6365322Z 
2023-01-11T21:52:40.6365641Z STAGE:2023-01-11 21:49:56 24481:24481 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6365957Z STAGE:2023-01-11 21:49:56 24482:24482 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6366277Z STAGE:2023-01-11 21:49:56 24482:24482 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6366609Z STAGE:2023-01-11 21:49:56 24481:24481 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6366957Z STAGE:2023-01-11 21:49:56 24482:24482 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6367303Z STAGE:2023-01-11 21:49:56 24481:24481 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6367404Z ok (4.132s)
2023-01-11T21:52:40.6367423Z 
2023-01-11T21:52:40.6367688Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6367793Z Ran 1 test in 4.132s
2023-01-11T21:52:40.6367813Z 
2023-01-11T21:52:40.6367902Z OK
2023-01-11T21:52:40.6367921Z 
2023-01-11T21:52:40.6368042Z Generating XML reports...
2023-01-11T21:52:40.6368484Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214952.xml
2023-01-11T21:52:40.6368857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6369032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6369409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6369601Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6369621Z 
2023-01-11T21:52:40.6369728Z Running tests...
2023-01-11T21:52:40.6369992Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6370303Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6370544Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2023-01-11T21:52:40.6370578Z 
2023-01-11T21:52:40.6370896Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6371017Z Ran 1 test in 0.002s
2023-01-11T21:52:40.6371037Z 
2023-01-11T21:52:40.6371142Z OK (skipped=1)
2023-01-11T21:52:40.6371209Z 
2023-01-11T21:52:40.6371334Z Generating XML reports...
2023-01-11T21:52:40.6371789Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214959.xml
2023-01-11T21:52:40.6372154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6372329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6372707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6372882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6372916Z 
2023-01-11T21:52:40.6373006Z Running tests...
2023-01-11T21:52:40.6373262Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6373575Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6373847Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2023-01-11T21:52:40.6373867Z 
2023-01-11T21:52:40.6374129Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6374239Z Ran 1 test in 0.002s
2023-01-11T21:52:40.6374258Z 
2023-01-11T21:52:40.6374361Z OK (skipped=1)
2023-01-11T21:52:40.6374380Z 
2023-01-11T21:52:40.6374501Z Generating XML reports...
2023-01-11T21:52:40.6374939Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215001.xml
2023-01-11T21:52:40.6375308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6375487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6375864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6376056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6376076Z 
2023-01-11T21:52:40.6376182Z Running tests...
2023-01-11T21:52:40.6376441Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6376751Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6376994Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6377213Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24660
2023-01-11T21:52:40.6377429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24661
2023-01-11T21:52:40.6377802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6377975Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6378355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6378549Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6378913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6379082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6379440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6379632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6379878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6380175Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6380587Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6381031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6381263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6381492Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6381730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.6381955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.6382354Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6382745Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6383082Z STAGE:2023-01-11 21:50:07 24661:24661 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6383405Z STAGE:2023-01-11 21:50:07 24660:24660 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6383736Z STAGE:2023-01-11 21:50:07 24660:24660 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6384059Z STAGE:2023-01-11 21:50:07 24661:24661 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6384406Z STAGE:2023-01-11 21:50:07 24660:24660 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6384751Z STAGE:2023-01-11 21:50:07 24661:24661 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6385065Z STAGE:2023-01-11 21:50:07 24660:24660 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6385382Z STAGE:2023-01-11 21:50:07 24661:24661 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6385718Z STAGE:2023-01-11 21:50:07 24661:24661 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6386271Z STAGE:2023-01-11 21:50:07 24661:24661 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:50:07 24660:24660 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6386292Z 
2023-01-11T21:52:40.6386631Z STAGE:2023-01-11 21:50:07 24660:24660 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6386731Z ok (4.260s)
2023-01-11T21:52:40.6386751Z 
2023-01-11T21:52:40.6387014Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6387121Z Ran 1 test in 4.260s
2023-01-11T21:52:40.6387140Z 
2023-01-11T21:52:40.6387234Z OK
2023-01-11T21:52:40.6387253Z 
2023-01-11T21:52:40.6387359Z Generating XML reports...
2023-01-11T21:52:40.6387811Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215003.xml
2023-01-11T21:52:40.6388189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6388363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6388741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6388932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6388952Z 
2023-01-11T21:52:40.6389055Z Running tests...
2023-01-11T21:52:40.6389317Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6389684Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6389931Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6390151Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24779
2023-01-11T21:52:40.6390425Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24780
2023-01-11T21:52:40.6390796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6390971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6391350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6391538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6391899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6392059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6392429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6392620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6392865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6393108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6393502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6393899Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6394130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6394361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6394500Z skip: Skipped due to small world size. (4.118s)
2023-01-11T21:52:40.6394523Z 
2023-01-11T21:52:40.6394788Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6394897Z Ran 1 test in 4.118s
2023-01-11T21:52:40.6394917Z 
2023-01-11T21:52:40.6395022Z OK (skipped=1)
2023-01-11T21:52:40.6395042Z 
2023-01-11T21:52:40.6395163Z Generating XML reports...
2023-01-11T21:52:40.6395611Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215010.xml
2023-01-11T21:52:40.6395982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6396218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6396648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6396826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6396846Z 
2023-01-11T21:52:40.6397023Z Running tests...
2023-01-11T21:52:40.6397434Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6397795Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6398094Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6398362Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24888
2023-01-11T21:52:40.6398565Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24889
2023-01-11T21:52:40.6398979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6416524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6417145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6417408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6417784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6417951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6418324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6418508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6418740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6418977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6419378Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6419769Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6419997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6420215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6420306Z ok (4.148s)
2023-01-11T21:52:40.6420329Z 
2023-01-11T21:52:40.6420585Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6420681Z Ran 1 test in 4.148s
2023-01-11T21:52:40.6420707Z 
2023-01-11T21:52:40.6420783Z OK
2023-01-11T21:52:40.6420802Z 
2023-01-11T21:52:40.6420915Z Generating XML reports...
2023-01-11T21:52:40.6421366Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215017.xml
2023-01-11T21:52:40.6421731Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6421900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6422273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6422452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6422472Z 
2023-01-11T21:52:40.6422570Z Running tests...
2023-01-11T21:52:40.6422815Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6423118Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6423354Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6423567Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 24997
2023-01-11T21:52:40.6423777Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 24998
2023-01-11T21:52:40.6424142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6424311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6424686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6424861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6425217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6425381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6425750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6425991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6426236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6426517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6426913Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6427302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6427519Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6427736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6427826Z ok (4.131s)
2023-01-11T21:52:40.6427846Z 
2023-01-11T21:52:40.6428101Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6428201Z Ran 1 test in 4.131s
2023-01-11T21:52:40.6428220Z 
2023-01-11T21:52:40.6428300Z OK
2023-01-11T21:52:40.6428320Z 
2023-01-11T21:52:40.6428436Z Generating XML reports...
2023-01-11T21:52:40.6428881Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215023.xml
2023-01-11T21:52:40.6429244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6429407Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6429779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6429958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6429978Z 
2023-01-11T21:52:40.6430075Z Running tests...
2023-01-11T21:52:40.6430329Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6430634Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6430888Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6431100Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25106
2023-01-11T21:52:40.6431303Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25107
2023-01-11T21:52:40.6431663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6431826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6432195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6432376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6432742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6432907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6433341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6433522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6433754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6433988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6434382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6434766Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6435042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6435261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6435392Z ok (4.242s)
2023-01-11T21:52:40.6435414Z 
2023-01-11T21:52:40.6435672Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6435768Z Ran 1 test in 4.242s
2023-01-11T21:52:40.6435802Z 
2023-01-11T21:52:40.6435877Z OK
2023-01-11T21:52:40.6435896Z 
2023-01-11T21:52:40.6436020Z Generating XML reports...
2023-01-11T21:52:40.6436473Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215030.xml
2023-01-11T21:52:40.6436845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6437022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6437407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6437602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6437625Z 
2023-01-11T21:52:40.6437730Z Running tests...
2023-01-11T21:52:40.6437977Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6438290Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6438582Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6438800Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25215
2023-01-11T21:52:40.6439017Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25216
2023-01-11T21:52:40.6439388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6439565Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6439944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6440122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6440495Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6440669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6441049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6441239Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6441485Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6441890Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6442131Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6442536Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6442751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6442972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6443309Z STAGE:2023-01-11 21:50:41 25215:25215 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6443637Z STAGE:2023-01-11 21:50:41 25216:25216 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6443971Z STAGE:2023-01-11 21:50:41 25215:25215 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6444614Z STAGE:2023-01-11 21:50:41 25215:25215 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6444980Z STAGE:2023-01-11 21:50:41 25216:25216 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6445391Z STAGE:2023-01-11 21:50:41 25216:25216 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6445489Z ok (4.255s)
2023-01-11T21:52:40.6445510Z 
2023-01-11T21:52:40.6445755Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6445866Z Ran 1 test in 4.255s
2023-01-11T21:52:40.6445886Z 
2023-01-11T21:52:40.6445976Z OK
2023-01-11T21:52:40.6445995Z 
2023-01-11T21:52:40.6446119Z Generating XML reports...
2023-01-11T21:52:40.6446575Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215037.xml
2023-01-11T21:52:40.6446952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6447128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6447506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6447696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6447716Z 
2023-01-11T21:52:40.6447808Z Running tests...
2023-01-11T21:52:40.6448071Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6448382Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6448667Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6448888Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25328
2023-01-11T21:52:40.6449111Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25329
2023-01-11T21:52:40.6449481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6449659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6450025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6450220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6450593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6450767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6451141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6451328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6451582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6451829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6452237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6452622Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6452853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6453080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6453420Z STAGE:2023-01-11 21:50:48 25329:25329 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6453748Z STAGE:2023-01-11 21:50:48 25328:25328 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6454133Z STAGE:2023-01-11 21:50:48 25328:25328 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6454701Z STAGE:2023-01-11 21:50:48 25329:25329 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:50:48 25328:25328 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6454765Z 
2023-01-11T21:52:40.6455123Z STAGE:2023-01-11 21:50:48 25329:25329 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6455225Z ok (4.241s)
2023-01-11T21:52:40.6455244Z 
2023-01-11T21:52:40.6455489Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6455603Z Ran 1 test in 4.241s
2023-01-11T21:52:40.6455622Z 
2023-01-11T21:52:40.6455713Z OK
2023-01-11T21:52:40.6455732Z 
2023-01-11T21:52:40.6455854Z Generating XML reports...
2023-01-11T21:52:40.6456317Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215044.xml
2023-01-11T21:52:40.6456696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6456876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6457260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6457450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6457470Z 
2023-01-11T21:52:40.6457561Z Running tests...
2023-01-11T21:52:40.6457821Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6458135Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6458410Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6458631Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25441
2023-01-11T21:52:40.6458850Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25442
2023-01-11T21:52:40.6459224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6459405Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6459770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6459960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6460329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6460504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6460890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6461075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6461325Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6461578Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6461981Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6462363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6462595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6462823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6463224Z STAGE:2023-01-11 21:50:54 25442:25442 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6463560Z STAGE:2023-01-11 21:50:54 25441:25441 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6463896Z STAGE:2023-01-11 21:50:54 25442:25442 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6464291Z STAGE:2023-01-11 21:50:54 25442:25442 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6464623Z STAGE:2023-01-11 21:50:54 25441:25441 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6464969Z STAGE:2023-01-11 21:50:54 25441:25441 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6465055Z ok (4.225s)
2023-01-11T21:52:40.6465075Z 
2023-01-11T21:52:40.6465333Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6465444Z Ran 1 test in 4.226s
2023-01-11T21:52:40.6465464Z 
2023-01-11T21:52:40.6465554Z OK
2023-01-11T21:52:40.6465574Z 
2023-01-11T21:52:40.6465700Z Generating XML reports...
2023-01-11T21:52:40.6466159Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215050.xml
2023-01-11T21:52:40.6466536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6466711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6467077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6467269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6467288Z 
2023-01-11T21:52:40.6467393Z Running tests...
2023-01-11T21:52:40.6467655Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6467967Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6468208Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2023-01-11T21:52:40.6468228Z 
2023-01-11T21:52:40.6468490Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6468604Z Ran 1 test in 0.002s
2023-01-11T21:52:40.6468623Z 
2023-01-11T21:52:40.6468731Z OK (skipped=1)
2023-01-11T21:52:40.6468750Z 
2023-01-11T21:52:40.6468856Z Generating XML reports...
2023-01-11T21:52:40.6469305Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215057.xml
2023-01-11T21:52:40.6469681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6469858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6470235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6470429Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6470449Z 
2023-01-11T21:52:40.6470557Z Running tests...
2023-01-11T21:52:40.6470815Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6471134Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6471386Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2023-01-11T21:52:40.6471406Z 
2023-01-11T21:52:40.6471661Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6471769Z Ran 1 test in 0.002s
2023-01-11T21:52:40.6471788Z 
2023-01-11T21:52:40.6471893Z OK (skipped=1)
2023-01-11T21:52:40.6471913Z 
2023-01-11T21:52:40.6472037Z Generating XML reports...
2023-01-11T21:52:40.6472486Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215100.xml
2023-01-11T21:52:40.6472909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6473088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6473502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6473688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6473708Z 
2023-01-11T21:52:40.6473815Z Running tests...
2023-01-11T21:52:40.6474073Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6474386Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6474649Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2023-01-11T21:52:40.6474668Z 
2023-01-11T21:52:40.6474924Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6475038Z Ran 1 test in 0.002s
2023-01-11T21:52:40.6475058Z 
2023-01-11T21:52:40.6475162Z OK (skipped=1)
2023-01-11T21:52:40.6475181Z 
2023-01-11T21:52:40.6475288Z Generating XML reports...
2023-01-11T21:52:40.6475744Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215102.xml
2023-01-11T21:52:40.6476117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6476294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6476674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6476863Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6476883Z 
2023-01-11T21:52:40.6476990Z Running tests...
2023-01-11T21:52:40.6477251Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6477565Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6477822Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6478042Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25653
2023-01-11T21:52:40.6478260Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25654
2023-01-11T21:52:40.6478632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6478810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6479194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6479388Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6479760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6479916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6480295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6480487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6480730Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6480978Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6481382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6481778Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6482054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6482288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6482661Z STAGE:2023-01-11 21:51:08 25654:25654 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6482980Z STAGE:2023-01-11 21:51:08 25653:25653 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6483311Z STAGE:2023-01-11 21:51:08 25654:25654 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6483659Z STAGE:2023-01-11 21:51:08 25654:25654 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6483992Z STAGE:2023-01-11 21:51:08 25653:25653 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6484578Z STAGE:2023-01-11 21:51:08 25653:25653 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6484687Z ok (4.246s)
2023-01-11T21:52:40.6484713Z 
2023-01-11T21:52:40.6484983Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6485093Z Ran 1 test in 4.246s
2023-01-11T21:52:40.6485116Z 
2023-01-11T21:52:40.6485193Z OK
2023-01-11T21:52:40.6485212Z 
2023-01-11T21:52:40.6485335Z Generating XML reports...
2023-01-11T21:52:40.6485790Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215104.xml
2023-01-11T21:52:40.6486160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6486334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6486715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6486904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6486924Z 
2023-01-11T21:52:40.6487033Z Running tests...
2023-01-11T21:52:40.6487280Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6487596Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6487859Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6488079Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25766
2023-01-11T21:52:40.6488297Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25767
2023-01-11T21:52:40.6488663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6488837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6489214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6489406Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6489762Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6489938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6490313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6490503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6490748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6490994Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6491397Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6491874Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6492119Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6492394Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6492494Z ok (4.095s)
2023-01-11T21:52:40.6492514Z 
2023-01-11T21:52:40.6492783Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6492894Z Ran 1 test in 4.096s
2023-01-11T21:52:40.6492913Z 
2023-01-11T21:52:40.6493004Z OK
2023-01-11T21:52:40.6493023Z 
2023-01-11T21:52:40.6493145Z Generating XML reports...
2023-01-11T21:52:40.6493600Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215111.xml
2023-01-11T21:52:40.6493973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6494137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6494514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6494705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6494724Z 
2023-01-11T21:52:40.6494833Z Running tests...
2023-01-11T21:52:40.6495090Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6495400Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6495685Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6495906Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25875
2023-01-11T21:52:40.6496107Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25876
2023-01-11T21:52:40.6496487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6496662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6497046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6497236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6497605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6497776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6498145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6498333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6498567Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6498813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6499211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6499611Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6499839Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6500068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6500402Z STAGE:2023-01-11 21:51:22 25876:25876 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6500726Z STAGE:2023-01-11 21:51:22 25875:25875 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6501150Z STAGE:2023-01-11 21:51:22 25875:25875 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6501494Z STAGE:2023-01-11 21:51:22 25875:25875 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6501874Z STAGE:2023-01-11 21:51:22 25876:25876 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6502221Z STAGE:2023-01-11 21:51:22 25876:25876 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6502321Z ok (4.211s)
2023-01-11T21:52:40.6502341Z 
2023-01-11T21:52:40.6502598Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6502705Z Ran 1 test in 4.211s
2023-01-11T21:52:40.6502725Z 
2023-01-11T21:52:40.6502812Z OK
2023-01-11T21:52:40.6502831Z 
2023-01-11T21:52:40.6502953Z Generating XML reports...
2023-01-11T21:52:40.6503391Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215118.xml
2023-01-11T21:52:40.6503771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6503947Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6504329Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6504522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6504541Z 
2023-01-11T21:52:40.6504649Z Running tests...
2023-01-11T21:52:40.6504910Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6505227Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6505510Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6505715Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 25988
2023-01-11T21:52:40.6505936Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 25989
2023-01-11T21:52:40.6506304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6506481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6506863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6507055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6507420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6507592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6507964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6508143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6508388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6508636Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6509033Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6509429Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6509659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6509885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6510216Z STAGE:2023-01-11 21:51:28 25988:25988 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6510568Z STAGE:2023-01-11 21:51:28 25989:25989 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T21:52:40.6510912Z STAGE:2023-01-11 21:51:28 25988:25988 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6511308Z STAGE:2023-01-11 21:51:28 25988:25988 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6511637Z STAGE:2023-01-11 21:51:28 25989:25989 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T21:52:40.6511979Z STAGE:2023-01-11 21:51:28 25989:25989 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T21:52:40.6512080Z ok (4.225s)
2023-01-11T21:52:40.6512101Z 
2023-01-11T21:52:40.6512361Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6512471Z Ran 1 test in 4.225s
2023-01-11T21:52:40.6512491Z 
2023-01-11T21:52:40.6512583Z OK
2023-01-11T21:52:40.6512603Z 
2023-01-11T21:52:40.6512709Z Generating XML reports...
2023-01-11T21:52:40.6513164Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215125.xml
2023-01-11T21:52:40.6513538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6513719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6514096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6514292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6514312Z 
2023-01-11T21:52:40.6514417Z Running tests...
2023-01-11T21:52:40.6514673Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6514983Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6515239Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6515458Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26101
2023-01-11T21:52:40.6515674Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26102
2023-01-11T21:52:40.6516048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6516223Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6516602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6516791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6517163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6517321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6517704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6517895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6518144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6518387Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6518791Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6519186Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6519415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6519642Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6519727Z ok (4.211s)
2023-01-11T21:52:40.6519799Z 
2023-01-11T21:52:40.6520074Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6520185Z Ran 1 test in 4.211s
2023-01-11T21:52:40.6520247Z 
2023-01-11T21:52:40.6520340Z OK
2023-01-11T21:52:40.6520359Z 
2023-01-11T21:52:40.6520482Z Generating XML reports...
2023-01-11T21:52:40.6520935Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215131.xml
2023-01-11T21:52:40.6521303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6521479Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6521844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6522037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6522056Z 
2023-01-11T21:52:40.6522167Z Running tests...
2023-01-11T21:52:40.6522429Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6522746Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6523021Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6523243Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26285
2023-01-11T21:52:40.6523460Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26286
2023-01-11T21:52:40.6523835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6523995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6524608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6524815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6525189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6525371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6525743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6525933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6526179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6526401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6526803Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6527200Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6527432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6527662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6527764Z ok (5.047s)
2023-01-11T21:52:40.6527783Z 
2023-01-11T21:52:40.6528048Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6528156Z Ran 1 test in 5.047s
2023-01-11T21:52:40.6528175Z 
2023-01-11T21:52:40.6528265Z OK
2023-01-11T21:52:40.6528284Z 
2023-01-11T21:52:40.6528391Z Generating XML reports...
2023-01-11T21:52:40.6528842Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215138.xml
2023-01-11T21:52:40.6529217Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6529481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6529877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6530125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6530145Z 
2023-01-11T21:52:40.6530249Z Running tests...
2023-01-11T21:52:40.6530510Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6530822Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6531075Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6531294Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26486
2023-01-11T21:52:40.6531510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26487
2023-01-11T21:52:40.6531887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6532062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6532443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6532634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6533003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6533204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6533592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6533778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6534028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6534273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6534671Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6535068Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6535296Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6535521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6535605Z ok (5.530s)
2023-01-11T21:52:40.6535625Z 
2023-01-11T21:52:40.6535890Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6536001Z Ran 1 test in 5.530s
2023-01-11T21:52:40.6536022Z 
2023-01-11T21:52:40.6536112Z OK
2023-01-11T21:52:40.6536132Z 
2023-01-11T21:52:40.6536253Z Generating XML reports...
2023-01-11T21:52:40.6536710Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215146.xml
2023-01-11T21:52:40.6537085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6537262Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6537630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6537822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6537842Z 
2023-01-11T21:52:40.6537948Z Running tests...
2023-01-11T21:52:40.6538209Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6538521Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6538839Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6539066Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26601
2023-01-11T21:52:40.6539328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26602
2023-01-11T21:52:40.6539706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6539864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6540244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6540434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6540804Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6540981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6541360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6541551Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6541796Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6542025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6542428Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6542827Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6543058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6543282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6543380Z ok (4.321s)
2023-01-11T21:52:40.6543401Z 
2023-01-11T21:52:40.6543660Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6543772Z Ran 1 test in 4.322s
2023-01-11T21:52:40.6543792Z 
2023-01-11T21:52:40.6543883Z OK
2023-01-11T21:52:40.6543902Z 
2023-01-11T21:52:40.6544007Z Generating XML reports...
2023-01-11T21:52:40.6544462Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215154.xml
2023-01-11T21:52:40.6544836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6545010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6545387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6545577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6545596Z 
2023-01-11T21:52:40.6545700Z Running tests...
2023-01-11T21:52:40.6545959Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6546258Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6546511Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6546730Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26714
2023-01-11T21:52:40.6546945Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26715
2023-01-11T21:52:40.6547312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6547489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6547921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6548120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6548490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6548696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6549069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6549252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6549498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6549741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6550148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6550542Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6550775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6551006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6551092Z ok (5.055s)
2023-01-11T21:52:40.6551111Z 
2023-01-11T21:52:40.6551374Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6551484Z Ran 1 test in 5.055s
2023-01-11T21:52:40.6551503Z 
2023-01-11T21:52:40.6551590Z OK
2023-01-11T21:52:40.6551609Z 
2023-01-11T21:52:40.6551731Z Generating XML reports...
2023-01-11T21:52:40.6552186Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215201.xml
2023-01-11T21:52:40.6552562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6552738Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6553107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6553295Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6553316Z 
2023-01-11T21:52:40.6553418Z Running tests...
2023-01-11T21:52:40.6553680Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6553996Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6554292Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6554511Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26825
2023-01-11T21:52:40.6554737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26826
2023-01-11T21:52:40.6555112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6555275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6555655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6555844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6556212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6556386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6556765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6556954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6557246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6557483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6557930Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6558331Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6558561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6558781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6559575Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:52:40.6560364Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T21:52:40.6560466Z ok (5.534s)
2023-01-11T21:52:40.6560487Z 
2023-01-11T21:52:40.6560754Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6560865Z Ran 1 test in 5.534s
2023-01-11T21:52:40.6560884Z 
2023-01-11T21:52:40.6560973Z OK
2023-01-11T21:52:40.6560995Z 
2023-01-11T21:52:40.6561116Z Generating XML reports...
2023-01-11T21:52:40.6561569Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215208.xml
2023-01-11T21:52:40.6561928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6562103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6562484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6562674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6562693Z 
2023-01-11T21:52:40.6562799Z Running tests...
2023-01-11T21:52:40.6563066Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6563380Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6563670Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6563873Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 26940
2023-01-11T21:52:40.6564091Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 26941
2023-01-11T21:52:40.6564643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6564818Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6565201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6565391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6565830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6566009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6566442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6566614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6566862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6567106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6567506Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6567912Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6568143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6568372Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6568617Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.6568852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.6569234Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6569622Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6569860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:52:40.6570107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:52:40.6570499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:52:40.6570895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:52:40.6570996Z ok (10.156s)
2023-01-11T21:52:40.6571017Z 
2023-01-11T21:52:40.6571283Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6571393Z Ran 1 test in 10.156s
2023-01-11T21:52:40.6571413Z 
2023-01-11T21:52:40.6571488Z OK
2023-01-11T21:52:40.6571507Z 
2023-01-11T21:52:40.6571629Z Generating XML reports...
2023-01-11T21:52:40.6572086Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215216.xml
2023-01-11T21:52:40.6572458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6572632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6573010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6573205Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6573225Z 
2023-01-11T21:52:40.6573328Z Running tests...
2023-01-11T21:52:40.6573575Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6573892Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T21:52:40.6574180Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T21:52:40.6574397Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27063
2023-01-11T21:52:40.6574614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27064
2023-01-11T21:52:40.6575052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6575237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6575663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6575849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6576203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T21:52:40.6576379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T21:52:40.6576754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T21:52:40.6576943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T21:52:40.6577188Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T21:52:40.6577433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T21:52:40.6577839Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6578230Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T21:52:40.6578461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T21:52:40.6578675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T21:52:40.6578915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T21:52:40.6579150Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T21:52:40.6579548Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6579936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T21:52:40.6580181Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T21:52:40.6580422Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T21:52:40.6580811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:52:40.6581210Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T21:52:40.6581295Z ok (10.045s)
2023-01-11T21:52:40.6581315Z 
2023-01-11T21:52:40.6581580Z ----------------------------------------------------------------------
2023-01-11T21:52:40.6581695Z Ran 1 test in 10.045s
2023-01-11T21:52:40.6581715Z 
2023-01-11T21:52:40.6581803Z OK
2023-01-11T21:52:40.6581823Z 
2023-01-11T21:52:40.6581939Z Generating XML reports...
2023-01-11T21:52:40.6582397Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215229.xml
2023-01-11T21:52:40.6582417Z 
2023-01-11T21:52:40.6582857Z ##[endgroup]
2023-01-11T21:52:40.6583315Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_p9kksj0a)
2023-01-11T21:52:40.6583351Z 
2023-01-11T21:52:40.6583548Z Running distributed tests for the gloo backend with file init_method in shard 1 of 3
2023-01-11T21:52:40.6584064Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_distributed_spawn.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 21:52:40.401213]
2023-01-11T22:21:33.2004919Z 
2023-01-11T22:21:33.2005745Z Expand the folded group to see the log file of distributed/test_distributed_spawn
2023-01-11T22:21:33.2007894Z ##[group]PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_fvmfkweh)
2023-01-11T22:21:33.2010697Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:21:33.2056180Z <unittest.suite.TestSuite tests=[<__main__.TestDistBackendWithSpawn testMethod=test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_3_level_hierarchical_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_Backend_enum_class>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallelCPU_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_2D_Input>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Channels_Last>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_No_Affine>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_non_default_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedDataParallel_with_amp_and_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_DistributedSampler_padding>, <__main__.TestDistBackendWithSpawn testMethod=test_SyncBatchNorm_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_allreduce_with_then_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_accumulate_gradients_no_sync_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_simple>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_coalesced_with_empty>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_cat_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_into_stack_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_default_pg>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_all_gather_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_max_complex_unsupported>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_coalesced_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_complex_unsupported_ops>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_multigpu_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_result_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_async>, <__main__.TestDistBackendWithSpawn testMethod=test_all_reduce_sum_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_equal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group>, <__main__.TestDistBackendWithSpawn testMethod=test_all_to_all_single_unequal_split_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_average_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_backend_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_full_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_group_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_global>, <__main__.TestDistBackendWithSpawn testMethod=test_barrier_timeout_group>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_gloo_tags>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_mixed_backend_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_no_rank_zero_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_op_list_err>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_ring_exchange_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_self_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_batch_isend_irecv_tensor_err>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_group>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_broadcast_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_compute_bucket_assignment_by_size_sparse_error_without_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_grad_as_bucket_view_false>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_apply_optim_in_backward_ignored_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_broadcast_buffer_via_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_buffer_hook_allreduce_return_future>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_build_debug_param_to_name_mapping_requires_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_comm_hook_logging>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_different_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_control_flow_same_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_create_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_device>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_forward_backward_hook>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_grad_div_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_allreduce_process_group>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_post_localSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_parity_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_pickling_powerSGD>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_ignore_params_arg>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_inference>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_join_model_equivalence>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_logging_data_gpu>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_num_params_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_model_diff_shape_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_err_ignore_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_multiple_nested_unused_params_error>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_namedtuple>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_new_tensor_in_fwd_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_profiling_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_python_error_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_returns_tensor_with_no_grad>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_shared_grad_acc_unused_params>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_static_graph_nested_types>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_bn_training_vs_eval>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_sync_module_states>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_input_join_disable>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_uneven_inputs_stop_iteration_sync_bn>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_unused_params_rebuild_buckets_exception>, <__main__.TestDistBackendWithSpawn testMethod=test_ddp_zero_output_features>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_destroy_group>, <__main__.TestDistBackendWithSpawn testMethod=test_detect_ddp_is_actually_static>, <__main__.TestDistBackendWithSpawn testMethod=test_different_graph_across_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_dump_DDP_relevant_env_vars>, <__main__.TestDistBackendWithSpawn testMethod=test_gather>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_group>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object>, <__main__.TestDistBackendWithSpawn testMethod=test_gather_object_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_get_backend>, <__main__.TestDistBackendWithSpawn testMethod=test_get_future>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_get_rank_size_group>, <__main__.TestDistBackendWithSpawn testMethod=test_invalid_static_graph>, <__main__.TestDistBackendWithSpawn testMethod=test_irecv>, <__main__.TestDistBackendWithSpawn testMethod=test_isend>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_isend_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_allreduce_hang_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_failure_order>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_rank_0_timeout>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_gloo_subgroup>, <__main__.TestDistBackendWithSpawn testMethod=test_monitored_barrier_wait_all_ranks>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allgather>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_allreduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_broadcast>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_backend_bool_reduce>, <__main__.TestDistBackendWithSpawn testMethod=test_nccl_high_priority_stream>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_input_rank_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_by_enumeration_negative_input_rank>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_group_size_exceeds_world_size>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_overlap_not_allowed>, <__main__.TestDistBackendWithSpawn testMethod=test_new_subgroups_world_size_not_divisible_by_group_size>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_dict_module>, <__main__.TestDistBackendWithSpawn testMethod=test_output_unused_in_loss_tuple_module>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager>, <__main__.TestDistBackendWithSpawn testMethod=test_periodic_model_averager_param_group>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view>, <__main__.TestDistBackendWithSpawn testMethod=test_post_localSGD_optimizer_step_reload>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_full_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_group_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_max>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_min>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_multigpu>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_product>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_tensor_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_scatter_v_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_cuda_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_reduce_sum_twice>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_checks>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_cuda_complex>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_full_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_group>, <__main__.TestDistBackendWithSpawn testMethod=test_scatter_object_list>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_any_source_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_nccl_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_autograd_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_send_recv_with_tag_torch_profiler>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum>, <__main__.TestDistBackendWithSpawn testMethod=test_sparse_all_reduce_sum_cuda>, <__main__.TestDistBackendWithSpawn testMethod=test_stateless_api_with_ddp>, <__main__.TestDistBackendWithSpawn testMethod=test_static_graph_api_cpu>, <__main__.TestDistBackendWithSpawn testMethod=test_sync_bn_logged>, <__main__.TestDistBackendWithSpawn testMethod=test_undefined_grad_parity_unused_parameters>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_with_logger>, <__main__.TestDistBackendWithSpawn testMethod=test_verify_model_across_rank_without_logger>]>
2023-01-11T22:21:33.2115969Z test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2116509Z test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2118517Z test_Backend_enum_class (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2118955Z test_DistributedDataParallel (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2119559Z test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2120049Z test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2120626Z test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2121139Z test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2121669Z test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2122214Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2122790Z test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2123335Z test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2123882Z test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2125016Z test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2125540Z test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2126050Z test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2126515Z test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2126965Z test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2127410Z test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2127874Z test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2128345Z test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2128837Z test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2129259Z test_all_gather (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2129651Z test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2130098Z test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2130529Z test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2130965Z test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2131380Z test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2131800Z test_all_gather_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2132198Z test_all_gather_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2132588Z test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2133000Z test_all_gather_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2133402Z test_all_gather_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2133815Z test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2134232Z test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2134659Z test_all_gather_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2135079Z test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2135485Z test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2135910Z test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2136317Z test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2136724Z test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2137180Z test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2137638Z test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2138188Z test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2138616Z test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2139144Z test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2139587Z test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2140021Z test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2140448Z test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2140901Z test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2141349Z test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2141751Z test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2142176Z test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2142616Z test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2143033Z test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2143455Z test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2143878Z test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2144300Z test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2144695Z test_all_reduce_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2145092Z test_all_reduce_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2145501Z test_all_reduce_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2145887Z test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2146284Z test_all_reduce_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2146667Z test_all_reduce_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2147043Z test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2147464Z test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2147888Z test_all_reduce_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2148296Z test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2148667Z test_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2149058Z test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2149465Z test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2149848Z test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2150252Z test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2150679Z test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2151075Z test_all_to_all (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2151445Z test_all_to_all_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2151834Z test_all_to_all_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2152206Z test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2152596Z test_all_to_all_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2152981Z test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2153366Z test_all_to_all_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2153729Z test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2154122Z test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2154548Z test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2154973Z test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2155404Z test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2155901Z test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2156364Z test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2156841Z test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2157278Z test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2157701Z test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2158135Z test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2158556Z test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2158997Z test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2159447Z test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2159899Z test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2160344Z test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2160788Z test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2161209Z test_average_parameters (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2161582Z test_backend_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2161953Z test_backend_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2162312Z test_barrier (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2162654Z test_barrier_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2163027Z test_barrier_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2163413Z test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2163787Z test_barrier_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2164161Z test_barrier_group_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2165177Z test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2165576Z test_barrier_timeout_global (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2165966Z test_barrier_timeout_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2166354Z test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2166752Z test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2167153Z test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2167555Z test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2167954Z test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2168359Z test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2168754Z test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2169163Z test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2169584Z test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2169976Z test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2170352Z test_broadcast (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2170716Z test_broadcast_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2171089Z test_broadcast_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2171469Z test_broadcast_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2171848Z test_broadcast_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2172237Z test_broadcast_object_list (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2172670Z test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2173268Z test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2173741Z test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2174239Z test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2174708Z test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2175135Z test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2175537Z test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2175938Z test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2176361Z test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2176803Z test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2177252Z test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2177691Z test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2178104Z test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2178572Z test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2178967Z test_ddp_create_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2179355Z test_ddp_device (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2179755Z test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2180155Z test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2180580Z test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2181024Z test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2181480Z test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2181888Z test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2182310Z test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2182780Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2183275Z test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2183863Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2184494Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2185119Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2185715Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2186340Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2186967Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2187576Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2188183Z test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2188726Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2189234Z test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2189745Z test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2190136Z test_ddp_inference (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2190594Z test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2191012Z test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2191423Z test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2191829Z test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2192285Z test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2192754Z test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2193225Z test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2193652Z test_ddp_namedtuple (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2194058Z test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2194488Z test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2194916Z test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2195362Z test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2195791Z test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2196194Z test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2196630Z test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2197070Z test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2197498Z test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2197897Z test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2198322Z test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2198753Z test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2199142Z test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2199582Z test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2200056Z test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2200483Z test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2200896Z test_destroy_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2201281Z test_destroy_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2201689Z test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2202101Z test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2202527Z test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2202919Z test_gather (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2203274Z test_gather_checks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2203649Z test_gather_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2204040Z test_gather_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2205000Z test_gather_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2205386Z test_gather_object (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2205780Z test_gather_object_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2206172Z test_get_backend (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2206525Z test_get_future (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2206886Z test_get_rank (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2207272Z test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2207657Z test_get_rank_size_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2208134Z test_invalid_static_graph (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2208521Z test_irecv (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2208855Z test_isend (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2209301Z test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2209711Z test_isend_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2210175Z test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2210620Z test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2211085Z test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2211507Z test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2211917Z test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2212357Z test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2212798Z test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2213210Z test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2213641Z test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2214058Z test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2214470Z test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2214869Z test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2215269Z test_new_subgroups (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2215676Z test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2216130Z test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2216632Z test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2217114Z test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2217570Z test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2218025Z test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2218486Z test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2218924Z test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2219330Z test_periodic_model_averager (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2219766Z test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2220210Z test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2220668Z test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2221137Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2221655Z test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2222147Z test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2222553Z test_reduce_full_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2222955Z test_reduce_full_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2223359Z test_reduce_full_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2223769Z test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2224145Z test_reduce_group_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2224530Z test_reduce_group_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2224926Z test_reduce_group_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2225301Z test_reduce_group_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2225733Z test_reduce_max (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2226112Z test_reduce_min (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2226476Z test_reduce_multigpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2226908Z test_reduce_product (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2227311Z test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2227721Z test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2228083Z test_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2228460Z test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2228849Z test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2229222Z test_reduce_sum_twice (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2229591Z test_scatter (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2229961Z test_scatter_checks (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2230333Z test_scatter_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2230712Z test_scatter_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2231107Z test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2231486Z test_scatter_full_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2231873Z test_scatter_group (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2232263Z test_scatter_object_list (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2232638Z test_send_recv (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2232998Z test_send_recv_any_source (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2233432Z test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2233889Z test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2234305Z test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2234711Z test_send_recv_nccl (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2235123Z test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2235541Z test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2235959Z test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2236356Z test_send_recv_with_tag (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2236773Z test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2237196Z test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2237612Z test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2238021Z test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2238417Z test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2238824Z test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2239213Z test_sync_bn_logged (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2239638Z test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2240080Z test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2240529Z test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2241273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2241719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2242308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2242785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2243019Z 
2023-01-11T22:21:33.2243133Z Running tests...
2023-01-11T22:21:33.2243579Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2244131Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2245487Z   test_1_level_hierarchical_model_averager_equivalent_to_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2246054Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27219
2023-01-11T22:21:33.2246510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27220
2023-01-11T22:21:33.2247140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2247600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2248165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2248643Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2249225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2249677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2250232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2250699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2251158Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2251649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2252313Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2253015Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2253532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2253994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2254509Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:21:33.2255342Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:21:33.2256004Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:21:33.2256812Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:21:33.2257476Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:21:33.2258303Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:21:33.2258971Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:21:33.2259772Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:21:33.2260436Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:21:33.2261006Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:21:33.2261930Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:21:33.2262835Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:21:33.2263579Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:21:33.2264407Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:21:33.2265063Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:Model averaging hierarchy:
2023-01-11T22:21:33.2265876Z INFO:torch.distributed.algorithms.model_averaging.hierarchical_model_averager:	Each group that has 2 processes average parameters every 4 iterations, if no higher-level averaging.
2023-01-11T22:21:33.2266373Z ok (5.605s)
2023-01-11T22:21:33.2266525Z 
2023-01-11T22:21:33.2266798Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2267141Z Ran 1 test in 5.605s
2023-01-11T22:21:33.2267285Z 
2023-01-11T22:21:33.2267380Z OK
2023-01-11T22:21:33.2267515Z 
2023-01-11T22:21:33.2267640Z Generating XML reports...
2023-01-11T22:21:33.2268255Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215244.xml
2023-01-11T22:21:33.2268970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2269430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2270013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2270488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2270726Z 
2023-01-11T22:21:33.2270818Z Running tests...
2023-01-11T22:21:33.2271225Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2271764Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2272286Z   test_3_level_hierarchical_model_averager (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.003s)
2023-01-11T22:21:33.2272597Z 
2023-01-11T22:21:33.2272860Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2273190Z Ran 1 test in 0.004s
2023-01-11T22:21:33.2273352Z 
2023-01-11T22:21:33.2273462Z OK (skipped=1)
2023-01-11T22:21:33.2273617Z 
2023-01-11T22:21:33.2273721Z Generating XML reports...
2023-01-11T22:21:33.2274333Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215252.xml
2023-01-11T22:21:33.2275059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2275521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2276081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2276556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2276788Z 
2023-01-11T22:21:33.2276897Z Running tests...
2023-01-11T22:21:33.2277284Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2277816Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2278332Z   test_Backend_enum_class (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2278828Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27364
2023-01-11T22:21:33.2279322Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27365
2023-01-11T22:21:33.2279943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2280447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2281012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2281495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2282076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2282527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2283080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2283550Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2284013Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2284866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2285542Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2286240Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2286772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2287231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2287583Z ok (4.250s)
2023-01-11T22:21:33.2287731Z 
2023-01-11T22:21:33.2288002Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2288338Z Ran 1 test in 4.250s
2023-01-11T22:21:33.2288480Z 
2023-01-11T22:21:33.2288582Z OK
2023-01-11T22:21:33.2288717Z 
2023-01-11T22:21:33.2288844Z Generating XML reports...
2023-01-11T22:21:33.2289461Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215255.xml
2023-01-11T22:21:33.2290172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2290628Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2291212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2291688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2291899Z 
2023-01-11T22:21:33.2292010Z Running tests...
2023-01-11T22:21:33.2292415Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2292957Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2293475Z   test_DistributedDataParallel (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2294537Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77317 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.643s)
2023-01-11T22:21:33.2295072Z 
2023-01-11T22:21:33.2295339Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2295668Z Ran 1 test in 1.643s
2023-01-11T22:21:33.2295831Z 
2023-01-11T22:21:33.2295938Z OK (skipped=1)
2023-01-11T22:21:33.2296074Z 
2023-01-11T22:21:33.2296201Z Generating XML reports...
2023-01-11T22:21:33.2296811Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215301.xml
2023-01-11T22:21:33.2297620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2298070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2298754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2299230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2299461Z 
2023-01-11T22:21:33.2299571Z Running tests...
2023-01-11T22:21:33.2299954Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2300484Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2301030Z   test_DistributedDataParallelCPU (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2301553Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27507
2023-01-11T22:21:33.2301996Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27508
2023-01-11T22:21:33.2302606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2303063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2303622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2304096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2304679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2305125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2305680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2306150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2306607Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2307099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2307761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2308455Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2308983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2309446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2309971Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2310471Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2310827Z ok (4.220s)
2023-01-11T22:21:33.2310959Z 
2023-01-11T22:21:33.2311233Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2311567Z Ran 1 test in 4.220s
2023-01-11T22:21:33.2311732Z 
2023-01-11T22:21:33.2311826Z OK
2023-01-11T22:21:33.2311960Z 
2023-01-11T22:21:33.2312066Z Generating XML reports...
2023-01-11T22:21:33.2312681Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215306.xml
2023-01-11T22:21:33.2313401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2313859Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2314421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2314951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2315189Z 
2023-01-11T22:21:33.2315301Z Running tests...
2023-01-11T22:21:33.2315689Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2316273Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2316837Z   test_DistributedDataParallelCPU_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2317377Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27620
2023-01-11T22:21:33.2317814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27621
2023-01-11T22:21:33.2318421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2318874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2319439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2319915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2320501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2320950Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2321507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2321976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2322434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2322942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2323589Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2324641Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2325197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2325658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2326143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2326638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2326995Z ok (4.231s)
2023-01-11T22:21:33.2327125Z 
2023-01-11T22:21:33.2327404Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2327736Z Ran 1 test in 4.231s
2023-01-11T22:21:33.2327898Z 
2023-01-11T22:21:33.2327993Z OK
2023-01-11T22:21:33.2328127Z 
2023-01-11T22:21:33.2328236Z Generating XML reports...
2023-01-11T22:21:33.2328852Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215312.xml
2023-01-11T22:21:33.2329577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2330035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2330598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2331071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2331304Z 
2023-01-11T22:21:33.2331418Z Running tests...
2023-01-11T22:21:33.2331804Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2332339Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2332973Z   test_DistributedDataParallel_SyncBatchNorm (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2333519Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27733
2023-01-11T22:21:33.2334018Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27734
2023-01-11T22:21:33.2334630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2335087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2335645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2336117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2336700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2337160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2337721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2338193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2338654Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2339161Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2339806Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2340501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2341035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2341502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2341990Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2342487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2342976Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2343440Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2343925Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2344409Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2344889Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2345348Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2345836Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2346316Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2346778Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2347253Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2347600Z ok (6.032s)
2023-01-11T22:21:33.2347748Z 
2023-01-11T22:21:33.2348024Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2348335Z Ran 1 test in 6.032s
2023-01-11T22:21:33.2348498Z 
2023-01-11T22:21:33.2348593Z OK
2023-01-11T22:21:33.2348727Z 
2023-01-11T22:21:33.2348853Z Generating XML reports...
2023-01-11T22:21:33.2349453Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215319.xml
2023-01-11T22:21:33.2350237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2350702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2351335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2351795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2352025Z 
2023-01-11T22:21:33.2352139Z Running tests...
2023-01-11T22:21:33.2352543Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2353062Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2353632Z   test_DistributedDataParallel_SyncBatchNorm_2D_Input (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2354174Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27848
2023-01-11T22:21:33.2354636Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27849
2023-01-11T22:21:33.2355231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2355688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2356269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2356725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2357308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2357755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2358324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2358777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2359240Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2359744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2360411Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2361090Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2361619Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2362101Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2362569Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2363062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2363550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2364033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2364650Z ok (5.160s)
2023-01-11T22:21:33.2364802Z 
2023-01-11T22:21:33.2365079Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2365410Z Ran 1 test in 5.160s
2023-01-11T22:21:33.2365572Z 
2023-01-11T22:21:33.2365647Z OK
2023-01-11T22:21:33.2365781Z 
2023-01-11T22:21:33.2365906Z Generating XML reports...
2023-01-11T22:21:33.2366520Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215328.xml
2023-01-11T22:21:33.2367242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2367755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2368354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2368893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2369127Z 
2023-01-11T22:21:33.2369217Z Running tests...
2023-01-11T22:21:33.2369626Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2370164Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2370748Z   test_DistributedDataParallel_SyncBatchNorm_Channels_Last (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2371281Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 27963
2023-01-11T22:21:33.2371739Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 27964
2023-01-11T22:21:33.2372356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2372815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2373376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2373855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2374438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2374867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2375440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2375906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2376366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2377019Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2377564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2378225Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2378758Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2379214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2379695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2380187Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2380658Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2381142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2381633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2382118Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2382587Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2383065Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2383412Z ok (5.136s)
2023-01-11T22:21:33.2383561Z 
2023-01-11T22:21:33.2383814Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2384146Z Ran 1 test in 5.137s
2023-01-11T22:21:33.2384309Z 
2023-01-11T22:21:33.2384408Z OK
2023-01-11T22:21:33.2384541Z 
2023-01-11T22:21:33.2384666Z Generating XML reports...
2023-01-11T22:21:33.2385314Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215335.xml
2023-01-11T22:21:33.2386051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2386574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2387146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2387621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2387854Z 
2023-01-11T22:21:33.2387964Z Running tests...
2023-01-11T22:21:33.2388369Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2388881Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2389485Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_Running_Value (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2390059Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28078
2023-01-11T22:21:33.2390498Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28079
2023-01-11T22:21:33.2391117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2391573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2392205Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2392667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2393259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2393709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2394289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2394741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2395202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2395708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2396356Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2397054Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2397579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2398059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2398530Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2399018Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2399377Z ok (5.466s)
2023-01-11T22:21:33.2399525Z 
2023-01-11T22:21:33.2399779Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2400108Z Ran 1 test in 5.466s
2023-01-11T22:21:33.2400269Z 
2023-01-11T22:21:33.2400364Z OK
2023-01-11T22:21:33.2400499Z 
2023-01-11T22:21:33.2400625Z Generating XML reports...
2023-01-11T22:21:33.2401220Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215343.xml
2023-01-11T22:21:33.2401946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2402403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2403017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2403503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2403785Z 
2023-01-11T22:21:33.2403896Z Running tests...
2023-01-11T22:21:33.2404634Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2405165Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2405759Z   test_DistributedDataParallel_SyncBatchNorm_Diff_Input_Sizes_gradient (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2406323Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28193
2023-01-11T22:21:33.2406785Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28194
2023-01-11T22:21:33.2407383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2407840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2408419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2408876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2409460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2409941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2410520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2410967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2411424Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2411937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2412582Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2413284Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2413816Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2414294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2414758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2415247Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2415731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2416223Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2416557Z ok (5.759s)
2023-01-11T22:21:33.2416703Z 
2023-01-11T22:21:33.2416975Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2417311Z Ran 1 test in 5.759s
2023-01-11T22:21:33.2417474Z 
2023-01-11T22:21:33.2417549Z OK
2023-01-11T22:21:33.2417683Z 
2023-01-11T22:21:33.2417807Z Generating XML reports...
2023-01-11T22:21:33.2418420Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215351.xml
2023-01-11T22:21:33.2419142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2419578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2420159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2420714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2420955Z 
2023-01-11T22:21:33.2421046Z Running tests...
2023-01-11T22:21:33.2421457Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2422056Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2422630Z   test_DistributedDataParallel_SyncBatchNorm_No_Affine (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2423156Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28308
2023-01-11T22:21:33.2423612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28309
2023-01-11T22:21:33.2424227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2424664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2425246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2425719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2426303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2426731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2427306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2427773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2428234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2428722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2429386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2430084Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2430594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2431068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2431546Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2432035Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2432503Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2432985Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2433334Z ok (5.625s)
2023-01-11T22:21:33.2433481Z 
2023-01-11T22:21:33.2433736Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2434066Z Ran 1 test in 5.625s
2023-01-11T22:21:33.2434228Z 
2023-01-11T22:21:33.2434327Z OK
2023-01-11T22:21:33.2434460Z 
2023-01-11T22:21:33.2434585Z Generating XML reports...
2023-01-11T22:21:33.2435181Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215400.xml
2023-01-11T22:21:33.2435903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2436359Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2436926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2437403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2437635Z 
2023-01-11T22:21:33.2437745Z Running tests...
2023-01-11T22:21:33.2438206Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2438734Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2439378Z   test_DistributedDataParallel_SyncBatchNorm_Single_Input_Per_Process (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2439949Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28423
2023-01-11T22:21:33.2440412Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28424
2023-01-11T22:21:33.2441009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2441467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2442045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2442505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2443086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2443541Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2444112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2444891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2445353Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2445860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2446513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2447216Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2447743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2448229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2448695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2449183Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2449670Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2450154Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2450485Z ok (5.168s)
2023-01-11T22:21:33.2450633Z 
2023-01-11T22:21:33.2450906Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2451235Z Ran 1 test in 5.168s
2023-01-11T22:21:33.2451402Z 
2023-01-11T22:21:33.2451478Z OK
2023-01-11T22:21:33.2451613Z 
2023-01-11T22:21:33.2451738Z Generating XML reports...
2023-01-11T22:21:33.2452353Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215408.xml
2023-01-11T22:21:33.2453079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2453516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2454094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2454570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2454802Z 
2023-01-11T22:21:33.2454893Z Running tests...
2023-01-11T22:21:33.2455296Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2455901Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2456474Z   test_DistributedDataParallel_non_default_stream (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2457592Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/76428 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.658s)
2023-01-11T22:21:33.2458120Z 
2023-01-11T22:21:33.2458385Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2458713Z Ran 1 test in 1.658s
2023-01-11T22:21:33.2458876Z 
2023-01-11T22:21:33.2458985Z OK (skipped=1)
2023-01-11T22:21:33.2459120Z 
2023-01-11T22:21:33.2459245Z Generating XML reports...
2023-01-11T22:21:33.2459863Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215416.xml
2023-01-11T22:21:33.2460582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2461024Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2461609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2462083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2462316Z 
2023-01-11T22:21:33.2462426Z Running tests...
2023-01-11T22:21:33.2462812Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2463343Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2463900Z   test_DistributedDataParallel_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2464436Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28572
2023-01-11T22:21:33.2464877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28573
2023-01-11T22:21:33.2465488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2465948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2466508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2466979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2467558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2468008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2468560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2469033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2469492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2469985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2470650Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2471344Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2471877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2472336Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2472679Z ok (4.214s)
2023-01-11T22:21:33.2472827Z 
2023-01-11T22:21:33.2473152Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2473490Z Ran 1 test in 4.214s
2023-01-11T22:21:33.2473634Z 
2023-01-11T22:21:33.2473729Z OK
2023-01-11T22:21:33.2473907Z 
2023-01-11T22:21:33.2474033Z Generating XML reports...
2023-01-11T22:21:33.2474653Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215420.xml
2023-01-11T22:21:33.2475357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2475812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2476393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2476868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2477079Z 
2023-01-11T22:21:33.2477190Z Running tests...
2023-01-11T22:21:33.2477596Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2478129Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2478687Z   test_DistributedDataParallel_with_amp_and_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2479761Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77294 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.633s)
2023-01-11T22:21:33.2480288Z 
2023-01-11T22:21:33.2480553Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2480884Z Ran 1 test in 1.634s
2023-01-11T22:21:33.2481046Z 
2023-01-11T22:21:33.2481136Z OK (skipped=1)
2023-01-11T22:21:33.2481292Z 
2023-01-11T22:21:33.2481417Z Generating XML reports...
2023-01-11T22:21:33.2482033Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215426.xml
2023-01-11T22:21:33.2482750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2483190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2483769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2484466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2484711Z 
2023-01-11T22:21:33.2484824Z Running tests...
2023-01-11T22:21:33.2485220Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2485756Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2486297Z   test_DistributedSampler_padding (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2486795Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28715
2023-01-11T22:21:33.2487250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28716
2023-01-11T22:21:33.2487871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2488328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2488885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2489358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2489940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2490368Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2491018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2491498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2492019Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2492508Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2493175Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2493876Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2494408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2494864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2495211Z ok (5.170s)
2023-01-11T22:21:33.2495363Z 
2023-01-11T22:21:33.2495631Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2495939Z Ran 1 test in 5.170s
2023-01-11T22:21:33.2496105Z 
2023-01-11T22:21:33.2496200Z OK
2023-01-11T22:21:33.2496333Z 
2023-01-11T22:21:33.2496459Z Generating XML reports...
2023-01-11T22:21:33.2497052Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215431.xml
2023-01-11T22:21:33.2497778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2498232Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2498811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2499267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2499500Z 
2023-01-11T22:21:33.2499612Z Running tests...
2023-01-11T22:21:33.2500016Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2500554Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2501046Z   test_SyncBatchNorm_process_group (__main__.TestDistBackendWithSpawn) ... skip: no torchvision (0.002s)
2023-01-11T22:21:33.2501336Z 
2023-01-11T22:21:33.2501600Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2501926Z Ran 1 test in 0.002s
2023-01-11T22:21:33.2502089Z 
2023-01-11T22:21:33.2502178Z OK (skipped=1)
2023-01-11T22:21:33.2502333Z 
2023-01-11T22:21:33.2502456Z Generating XML reports...
2023-01-11T22:21:33.2503060Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215438.xml
2023-01-11T22:21:33.2503786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2504222Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2504802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2505280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2505511Z 
2023-01-11T22:21:33.2505601Z Running tests...
2023-01-11T22:21:33.2506003Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2506534Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2507003Z   test_accumulate_gradients_no_sync (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2507470Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2507957Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28859
2023-01-11T22:21:33.2508470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28860
2023-01-11T22:21:33.2509074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2509593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2510221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2510699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2511261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2511710Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2512283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2512757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2513195Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2513706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2514368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2515047Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2515575Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2516052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2516399Z ok (4.215s)
2023-01-11T22:21:33.2516528Z 
2023-01-11T22:21:33.2516795Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2517125Z Ran 1 test in 4.215s
2023-01-11T22:21:33.2517288Z 
2023-01-11T22:21:33.2517385Z OK
2023-01-11T22:21:33.2517520Z 
2023-01-11T22:21:33.2517626Z Generating XML reports...
2023-01-11T22:21:33.2518237Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215441.xml
2023-01-11T22:21:33.2518957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2519411Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2519972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2520440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2520670Z 
2023-01-11T22:21:33.2520780Z Running tests...
2023-01-11T22:21:33.2521163Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2521697Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2522184Z   test_accumulate_gradients_no_sync_allreduce_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2522693Z Runs multiple iterations on _test_accumulate_gradients_no_sync ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2523166Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 28972
2023-01-11T22:21:33.2523621Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 28973
2023-01-11T22:21:33.2524536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2524992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2525586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2526141Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2526736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2527227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2527803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2528273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2528732Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2529219Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2529880Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2530581Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2531094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2531581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2532063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2532557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2532896Z ok (4.221s)
2023-01-11T22:21:33.2533042Z 
2023-01-11T22:21:33.2533315Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2533644Z Ran 1 test in 4.221s
2023-01-11T22:21:33.2533807Z 
2023-01-11T22:21:33.2533882Z OK
2023-01-11T22:21:33.2534014Z 
2023-01-11T22:21:33.2534139Z Generating XML reports...
2023-01-11T22:21:33.2534756Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215448.xml
2023-01-11T22:21:33.2535478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2535917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2536493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2536967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2537198Z 
2023-01-11T22:21:33.2537308Z Running tests...
2023-01-11T22:21:33.2537694Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2538225Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2538731Z   test_accumulate_gradients_no_sync_allreduce_with_then_hook (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2539260Z Runs multiple iterations on _test_accumulate_gradients_no_sync using allreduce ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2539772Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29085
2023-01-11T22:21:33.2540232Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29086
2023-01-11T22:21:33.2540847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2541280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2541854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2542328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2542890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2543341Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2543964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2544440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2544926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2545432Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2546099Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2546795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2547305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2547790Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2548276Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2548748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.2549108Z ok (4.225s)
2023-01-11T22:21:33.2549258Z 
2023-01-11T22:21:33.2549530Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2549861Z Ran 1 test in 4.226s
2023-01-11T22:21:33.2550004Z 
2023-01-11T22:21:33.2550100Z OK
2023-01-11T22:21:33.2550234Z 
2023-01-11T22:21:33.2550358Z Generating XML reports...
2023-01-11T22:21:33.2550969Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215454.xml
2023-01-11T22:21:33.2551671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2552132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2552718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2553196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2553412Z 
2023-01-11T22:21:33.2553522Z Running tests...
2023-01-11T22:21:33.2553928Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2554462Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2554925Z   test_accumulate_gradients_no_sync_grad_is_view (__main__.TestDistBackendWithSpawn)
2023-01-11T22:21:33.2555428Z Runs _test_accumulate_gradients_no_sync using default inputs ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2555918Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29198
2023-01-11T22:21:33.2556377Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29199
2023-01-11T22:21:33.2556973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2557431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2558014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2558470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2559051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2559500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2560076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2560529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2561040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2561557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2562270Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2562950Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2563482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2563963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2564529Z ok (4.256s)
2023-01-11T22:21:33.2564683Z 
2023-01-11T22:21:33.2564960Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2565291Z Ran 1 test in 4.256s
2023-01-11T22:21:33.2565455Z 
2023-01-11T22:21:33.2565552Z OK
2023-01-11T22:21:33.2565673Z 
2023-01-11T22:21:33.2565799Z Generating XML reports...
2023-01-11T22:21:33.2566415Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215501.xml
2023-01-11T22:21:33.2567147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2567586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2568170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2568640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2568874Z 
2023-01-11T22:21:33.2568985Z Running tests...
2023-01-11T22:21:33.2569370Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2569906Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2570410Z   test_all_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2570873Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29311
2023-01-11T22:21:33.2571336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29312
2023-01-11T22:21:33.2571946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2572400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2572957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2573431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2574013Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2574446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2575015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2575487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2575943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2576430Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2577093Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2577787Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2578323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2578892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2579490Z STAGE:2023-01-11 21:55:12 29311:29311 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2580130Z STAGE:2023-01-11 21:55:12 29312:29312 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2580934Z STAGE:2023-01-11 21:55:12 29312:29312 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:55:12 29311:29311 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2581306Z 
2023-01-11T22:21:33.2581881Z STAGE:2023-01-11 21:55:12 29311:29311 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:55:12 29312:29312 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2582289Z 
2023-01-11T22:21:33.2582619Z STAGE:2023-01-11 21:55:12 29312:29312 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2583198Z STAGE:2023-01-11 21:55:12 29311:29311 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2583784Z STAGE:2023-01-11 21:55:12 29311:29311 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2584348Z STAGE:2023-01-11 21:55:12 29312:29312 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2584946Z STAGE:2023-01-11 21:55:12 29311:29311 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2585554Z STAGE:2023-01-11 21:55:12 29312:29312 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2585912Z ok (4.207s)
2023-01-11T22:21:33.2586042Z 
2023-01-11T22:21:33.2586310Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2586640Z Ran 1 test in 4.207s
2023-01-11T22:21:33.2586803Z 
2023-01-11T22:21:33.2586898Z OK
2023-01-11T22:21:33.2587035Z 
2023-01-11T22:21:33.2587141Z Generating XML reports...
2023-01-11T22:21:33.2587757Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215508.xml
2023-01-11T22:21:33.2588483Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2588945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2589510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2589983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2590214Z 
2023-01-11T22:21:33.2590326Z Running tests...
2023-01-11T22:21:33.2590712Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2591250Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2591784Z   test_all_gather_coalesced_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2592297Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29424
2023-01-11T22:21:33.2592734Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29425
2023-01-11T22:21:33.2593344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2593800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2594359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2594835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2595419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2595869Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2596479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2596960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2597473Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2597982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2598628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2599321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2599849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2600309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2601105Z STAGE:2023-01-11 21:55:18 29425:29425 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:55:18 29424:29424 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2601489Z 
2023-01-11T22:21:33.2602238Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2602876Z   warnings.warn(
2023-01-11T22:21:33.2603755Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2604579Z   warnings.warn(
2023-01-11T22:21:33.2605291Z STAGE:2023-01-11 21:55:18 29425:29425 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:55:18 29424:29424 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2605688Z 
2023-01-11T22:21:33.2606260Z STAGE:2023-01-11 21:55:18 29424:29424 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:55:18 29425:29425 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2606666Z 
2023-01-11T22:21:33.2607196Z STAGE:2023-01-11 21:55:18 29424:29424 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:55:18 29425:29425 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2607576Z 
2023-01-11T22:21:33.2607914Z STAGE:2023-01-11 21:55:18 29424:29424 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2608487Z STAGE:2023-01-11 21:55:18 29425:29425 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2609090Z STAGE:2023-01-11 21:55:18 29424:29424 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2609706Z STAGE:2023-01-11 21:55:18 29425:29425 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2610333Z STAGE:2023-01-11 21:55:18 29424:29424 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2610885Z STAGE:2023-01-11 21:55:18 29425:29425 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2611470Z STAGE:2023-01-11 21:55:18 29425:29425 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2612049Z STAGE:2023-01-11 21:55:18 29424:29424 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2612620Z STAGE:2023-01-11 21:55:18 29425:29425 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2613225Z STAGE:2023-01-11 21:55:18 29424:29424 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2613665Z ok (4.232s)
2023-01-11T22:21:33.2613827Z 
2023-01-11T22:21:33.2614100Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2614473Z Ran 1 test in 4.232s
2023-01-11T22:21:33.2614637Z 
2023-01-11T22:21:33.2614732Z OK
2023-01-11T22:21:33.2614865Z 
2023-01-11T22:21:33.2614990Z Generating XML reports...
2023-01-11T22:21:33.2615592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215515.xml
2023-01-11T22:21:33.2616316Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2616776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2617356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2617816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2618050Z 
2023-01-11T22:21:33.2618161Z Running tests...
2023-01-11T22:21:33.2618568Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2618893Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2619179Z   test_all_gather_coalesced_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2619383Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29537
2023-01-11T22:21:33.2619604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29538
2023-01-11T22:21:33.2619981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2620160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2620545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2620740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2621107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2621287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2621645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2621837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2622086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2622334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2622739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2623141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2623377Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2623613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2623856Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.2624080Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.2624484Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2624880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2625265Z STAGE:2023-01-11 21:55:25 29538:29538 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2625602Z STAGE:2023-01-11 21:55:25 29537:29537 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2626414Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2626529Z   warnings.warn(
2023-01-11T22:21:33.2627271Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2627386Z   warnings.warn(
2023-01-11T22:21:33.2627943Z STAGE:2023-01-11 21:55:25 29538:29538 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:55:25 29537:29537 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2627964Z 
2023-01-11T22:21:33.2628540Z STAGE:2023-01-11 21:55:25 29537:29537 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:55:25 29538:29538 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2628564Z 
2023-01-11T22:21:33.2628892Z STAGE:2023-01-11 21:55:25 29538:29538 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2629195Z STAGE:2023-01-11 21:55:25 29537:29537 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2629529Z STAGE:2023-01-11 21:55:25 29537:29537 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2629864Z STAGE:2023-01-11 21:55:25 29538:29538 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2630215Z STAGE:2023-01-11 21:55:25 29537:29537 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2630564Z STAGE:2023-01-11 21:55:25 29538:29538 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2630897Z STAGE:2023-01-11 21:55:25 29537:29537 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2631220Z STAGE:2023-01-11 21:55:25 29538:29538 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2631554Z STAGE:2023-01-11 21:55:25 29537:29537 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2631882Z STAGE:2023-01-11 21:55:25 29538:29538 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2632211Z STAGE:2023-01-11 21:55:25 29537:29537 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2632556Z STAGE:2023-01-11 21:55:25 29538:29538 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2632659Z ok (4.314s)
2023-01-11T22:21:33.2632682Z 
2023-01-11T22:21:33.2632948Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2633061Z Ran 1 test in 4.314s
2023-01-11T22:21:33.2633084Z 
2023-01-11T22:21:33.2633179Z OK
2023-01-11T22:21:33.2633197Z 
2023-01-11T22:21:33.2633323Z Generating XML reports...
2023-01-11T22:21:33.2633781Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215521.xml
2023-01-11T22:21:33.2634138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2634316Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2634700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2634896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2634915Z 
2023-01-11T22:21:33.2635080Z Running tests...
2023-01-11T22:21:33.2635353Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2635671Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2635998Z   test_all_gather_coalesced_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2636223Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29656
2023-01-11T22:21:33.2636425Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29657
2023-01-11T22:21:33.2636801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2636977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2637360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2637558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2637921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2638099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2638474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2638646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2638896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2639144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2639550Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2639951Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2640186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2640422Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2640584Z skip: Skipped due to small world size. (4.218s)
2023-01-11T22:21:33.2640603Z 
2023-01-11T22:21:33.2640873Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2640966Z Ran 1 test in 4.218s
2023-01-11T22:21:33.2640985Z 
2023-01-11T22:21:33.2641094Z OK (skipped=1)
2023-01-11T22:21:33.2641113Z 
2023-01-11T22:21:33.2641238Z Generating XML reports...
2023-01-11T22:21:33.2641693Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215528.xml
2023-01-11T22:21:33.2642070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2642248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2642630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2642827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2642847Z 
2023-01-11T22:21:33.2642956Z Running tests...
2023-01-11T22:21:33.2643200Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2643517Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2643796Z   test_all_gather_coalesced_simple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2644019Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29765
2023-01-11T22:21:33.2644471Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29766
2023-01-11T22:21:33.2644938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2645125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2645572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2645746Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2646116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2646293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2646668Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2646859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2647113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2647365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2647770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2648173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2648388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2648618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2648961Z STAGE:2023-01-11 21:55:39 29766:29766 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2649290Z STAGE:2023-01-11 21:55:39 29765:29765 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2650043Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2650163Z   warnings.warn(
2023-01-11T22:21:33.2650905Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2651018Z   warnings.warn(
2023-01-11T22:21:33.2651361Z STAGE:2023-01-11 21:55:39 29766:29766 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2651692Z STAGE:2023-01-11 21:55:39 29765:29765 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2652025Z STAGE:2023-01-11 21:55:39 29766:29766 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2652372Z STAGE:2023-01-11 21:55:39 29765:29765 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2652713Z STAGE:2023-01-11 21:55:39 29766:29766 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2653037Z STAGE:2023-01-11 21:55:39 29765:29765 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2653372Z STAGE:2023-01-11 21:55:39 29765:29765 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2653701Z STAGE:2023-01-11 21:55:39 29766:29766 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2654274Z STAGE:2023-01-11 21:55:39 29765:29765 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:55:39 29766:29766 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2654294Z 
2023-01-11T22:21:33.2654679Z STAGE:2023-01-11 21:55:39 29766:29766 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2655011Z STAGE:2023-01-11 21:55:39 29765:29765 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2655400Z STAGE:2023-01-11 21:55:39 29766:29766 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2655709Z STAGE:2023-01-11 21:55:39 29765:29765 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2656056Z STAGE:2023-01-11 21:55:39 29766:29766 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2656406Z STAGE:2023-01-11 21:55:39 29765:29765 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2656509Z ok (4.233s)
2023-01-11T22:21:33.2656528Z 
2023-01-11T22:21:33.2656791Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2656903Z Ran 1 test in 4.233s
2023-01-11T22:21:33.2656924Z 
2023-01-11T22:21:33.2657021Z OK
2023-01-11T22:21:33.2657041Z 
2023-01-11T22:21:33.2657167Z Generating XML reports...
2023-01-11T22:21:33.2657604Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215535.xml
2023-01-11T22:21:33.2657981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2658158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2658544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2658737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2658757Z 
2023-01-11T22:21:33.2658865Z Running tests...
2023-01-11T22:21:33.2659128Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2659449Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2659734Z   test_all_gather_coalesced_with_empty (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2659941Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29878
2023-01-11T22:21:33.2660163Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29879
2023-01-11T22:21:33.2660540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2660717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2661099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2661294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2661664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2661840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2662201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2662397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2662646Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2662891Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2663296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2663698Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2663984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2664223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2664565Z STAGE:2023-01-11 21:55:46 29878:29878 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2664919Z STAGE:2023-01-11 21:55:46 29879:29879 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2665663Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2665780Z   warnings.warn(
2023-01-11T22:21:33.2666524Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2588: UserWarning: torch.distributed.all_gather_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2666638Z   warnings.warn(
2023-01-11T22:21:33.2666977Z STAGE:2023-01-11 21:55:46 29878:29878 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2667312Z STAGE:2023-01-11 21:55:46 29879:29879 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2667664Z STAGE:2023-01-11 21:55:46 29878:29878 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2668012Z STAGE:2023-01-11 21:55:46 29879:29879 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2668114Z ok (4.230s)
2023-01-11T22:21:33.2668135Z 
2023-01-11T22:21:33.2668379Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2668492Z Ran 1 test in 4.230s
2023-01-11T22:21:33.2668512Z 
2023-01-11T22:21:33.2668606Z OK
2023-01-11T22:21:33.2668625Z 
2023-01-11T22:21:33.2668750Z Generating XML reports...
2023-01-11T22:21:33.2669207Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215542.xml
2023-01-11T22:21:33.2669583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2669766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2670149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2670322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2670361Z 
2023-01-11T22:21:33.2670451Z Running tests...
2023-01-11T22:21:33.2670715Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2671032Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2671301Z   test_all_gather_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2671521Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 29991
2023-01-11T22:21:33.2671742Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 29992
2023-01-11T22:21:33.2672119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2672296Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2672660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2672852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2673216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2673392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2673817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2674014Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2674306Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2674553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2674942Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2675343Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2675578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2675811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2676152Z STAGE:2023-01-11 21:55:52 29991:29991 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2676480Z STAGE:2023-01-11 21:55:52 29992:29992 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2676821Z STAGE:2023-01-11 21:55:52 29991:29991 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2677155Z STAGE:2023-01-11 21:55:52 29992:29992 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2677504Z STAGE:2023-01-11 21:55:52 29991:29991 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2677832Z STAGE:2023-01-11 21:55:52 29992:29992 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2678162Z STAGE:2023-01-11 21:55:52 29991:29991 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2678492Z STAGE:2023-01-11 21:55:52 29992:29992 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2678830Z STAGE:2023-01-11 21:55:52 29991:29991 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2679156Z STAGE:2023-01-11 21:55:52 29992:29992 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2679507Z STAGE:2023-01-11 21:55:52 29991:29991 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2679855Z STAGE:2023-01-11 21:55:52 29992:29992 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2679959Z ok (4.222s)
2023-01-11T22:21:33.2679979Z 
2023-01-11T22:21:33.2680244Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2680338Z Ran 1 test in 4.223s
2023-01-11T22:21:33.2680358Z 
2023-01-11T22:21:33.2680451Z OK
2023-01-11T22:21:33.2680470Z 
2023-01-11T22:21:33.2680597Z Generating XML reports...
2023-01-11T22:21:33.2681059Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215548.xml
2023-01-11T22:21:33.2681432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2681613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2681997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2682194Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2682214Z 
2023-01-11T22:21:33.2682323Z Running tests...
2023-01-11T22:21:33.2682569Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2682887Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2683150Z   test_all_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2023-01-11T22:21:33.2683170Z 
2023-01-11T22:21:33.2683479Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2683596Z Ran 1 test in 0.002s
2023-01-11T22:21:33.2683615Z 
2023-01-11T22:21:33.2683724Z OK (skipped=1)
2023-01-11T22:21:33.2683783Z 
2023-01-11T22:21:33.2683910Z Generating XML reports...
2023-01-11T22:21:33.2684587Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215555.xml
2023-01-11T22:21:33.2684956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2685138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2685523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2685717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2685736Z 
2023-01-11T22:21:33.2685846Z Running tests...
2023-01-11T22:21:33.2686112Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2686430Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2686714Z   test_all_gather_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all gather (0.002s)
2023-01-11T22:21:33.2686733Z 
2023-01-11T22:21:33.2686996Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2687089Z Ran 1 test in 0.002s
2023-01-11T22:21:33.2687109Z 
2023-01-11T22:21:33.2687218Z OK (skipped=1)
2023-01-11T22:21:33.2687237Z 
2023-01-11T22:21:33.2687362Z Generating XML reports...
2023-01-11T22:21:33.2687815Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215558.xml
2023-01-11T22:21:33.2688189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2688369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2688752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2688949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2688969Z 
2023-01-11T22:21:33.2689079Z Running tests...
2023-01-11T22:21:33.2689320Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2689635Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2689903Z   test_all_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2690125Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30170
2023-01-11T22:21:33.2690348Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30171
2023-01-11T22:21:33.2690726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2690903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2691287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2691461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2691831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2692006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2692435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2692627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2692879Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2693200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2693619Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2694093Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2694309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2694542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2694784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.2695028Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.2695434Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2695831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2696170Z STAGE:2023-01-11 21:56:04 30171:30171 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2696496Z STAGE:2023-01-11 21:56:04 30170:30170 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2696832Z STAGE:2023-01-11 21:56:04 30171:30171 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2697143Z STAGE:2023-01-11 21:56:04 30170:30170 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2697494Z STAGE:2023-01-11 21:56:04 30171:30171 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2697844Z STAGE:2023-01-11 21:56:04 30170:30170 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2698176Z STAGE:2023-01-11 21:56:04 30171:30171 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2698501Z STAGE:2023-01-11 21:56:04 30170:30170 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2698841Z STAGE:2023-01-11 21:56:04 30171:30171 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2699177Z STAGE:2023-01-11 21:56:04 30170:30170 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2699524Z STAGE:2023-01-11 21:56:04 30171:30171 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2699867Z STAGE:2023-01-11 21:56:04 30170:30170 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2699951Z ok (4.355s)
2023-01-11T22:21:33.2699971Z 
2023-01-11T22:21:33.2700238Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2700351Z Ran 1 test in 4.355s
2023-01-11T22:21:33.2700370Z 
2023-01-11T22:21:33.2700467Z OK
2023-01-11T22:21:33.2700487Z 
2023-01-11T22:21:33.2700612Z Generating XML reports...
2023-01-11T22:21:33.2701069Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215600.xml
2023-01-11T22:21:33.2701449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2701627Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2701992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2702187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2702206Z 
2023-01-11T22:21:33.2702316Z Running tests...
2023-01-11T22:21:33.2702582Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2702947Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2703215Z   test_all_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2703481Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30289
2023-01-11T22:21:33.2703702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30290
2023-01-11T22:21:33.2704082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2704240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2704623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2704815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2705182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2705362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2705737Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2705931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2706180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2706408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2706811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2707209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2707444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2707680Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2707844Z skip: Skipped due to small world size. (4.246s)
2023-01-11T22:21:33.2707867Z 
2023-01-11T22:21:33.2708137Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2708251Z Ran 1 test in 4.246s
2023-01-11T22:21:33.2708270Z 
2023-01-11T22:21:33.2708378Z OK (skipped=1)
2023-01-11T22:21:33.2708397Z 
2023-01-11T22:21:33.2708503Z Generating XML reports...
2023-01-11T22:21:33.2708958Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215607.xml
2023-01-11T22:21:33.2709333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2709512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2709892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2710129Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2710149Z 
2023-01-11T22:21:33.2710263Z Running tests...
2023-01-11T22:21:33.2710528Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2710843Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2711127Z   test_all_gather_into_cat_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2023-01-11T22:21:33.2711147Z 
2023-01-11T22:21:33.2711411Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2711524Z Ran 1 test in 0.002s
2023-01-11T22:21:33.2711543Z 
2023-01-11T22:21:33.2711652Z OK (skipped=1)
2023-01-11T22:21:33.2711671Z 
2023-01-11T22:21:33.2711794Z Generating XML reports...
2023-01-11T22:21:33.2712300Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215614.xml
2023-01-11T22:21:33.2712685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2712910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2713295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2713471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2713491Z 
2023-01-11T22:21:33.2713602Z Running tests...
2023-01-11T22:21:33.2713861Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2714175Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2714481Z   test_all_gather_into_stack_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_gather_into_tensor (0.002s)
2023-01-11T22:21:33.2714502Z 
2023-01-11T22:21:33.2714764Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2714877Z Ran 1 test in 0.002s
2023-01-11T22:21:33.2714901Z 
2023-01-11T22:21:33.2715009Z OK (skipped=1)
2023-01-11T22:21:33.2715028Z 
2023-01-11T22:21:33.2715151Z Generating XML reports...
2023-01-11T22:21:33.2715585Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215616.xml
2023-01-11T22:21:33.2715955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2716131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2716512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2716705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2716728Z 
2023-01-11T22:21:33.2716838Z Running tests...
2023-01-11T22:21:33.2717103Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2717424Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2717695Z   test_all_gather_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2023-01-11T22:21:33.2717734Z 
2023-01-11T22:21:33.2717974Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2718086Z Ran 1 test in 0.002s
2023-01-11T22:21:33.2718106Z 
2023-01-11T22:21:33.2718213Z OK (skipped=1)
2023-01-11T22:21:33.2718232Z 
2023-01-11T22:21:33.2718359Z Generating XML reports...
2023-01-11T22:21:33.2718808Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215618.xml
2023-01-11T22:21:33.2719185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2719363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2719745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2719924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2719962Z 
2023-01-11T22:21:33.2720054Z Running tests...
2023-01-11T22:21:33.2720317Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2720633Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2720935Z   test_all_gather_multigpu_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports allgather multigpu (0.002s)
2023-01-11T22:21:33.2720955Z 
2023-01-11T22:21:33.2721218Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2721382Z Ran 1 test in 0.002s
2023-01-11T22:21:33.2721403Z 
2023-01-11T22:21:33.2721516Z OK (skipped=1)
2023-01-11T22:21:33.2721535Z 
2023-01-11T22:21:33.2721660Z Generating XML reports...
2023-01-11T22:21:33.2722142Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215621.xml
2023-01-11T22:21:33.2722513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2722692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2723074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2723267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2723287Z 
2023-01-11T22:21:33.2723398Z Running tests...
2023-01-11T22:21:33.2723661Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2723981Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2724457Z   test_all_gather_object_default_pg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2724697Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30530
2023-01-11T22:21:33.2724918Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30531
2023-01-11T22:21:33.2725302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2725480Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2725863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2726058Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2726431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2726606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2726967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2727160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2727410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2727656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2728061Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2728461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2728699Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2728934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2729040Z ok (4.265s)
2023-01-11T22:21:33.2729060Z 
2023-01-11T22:21:33.2729309Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2729422Z Ran 1 test in 4.265s
2023-01-11T22:21:33.2729442Z 
2023-01-11T22:21:33.2729535Z OK
2023-01-11T22:21:33.2729555Z 
2023-01-11T22:21:33.2729681Z Generating XML reports...
2023-01-11T22:21:33.2730132Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215623.xml
2023-01-11T22:21:33.2730505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2730684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2731144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2731326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2731416Z 
2023-01-11T22:21:33.2731511Z Running tests...
2023-01-11T22:21:33.2731775Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2732092Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2732368Z   test_all_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2732591Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30639
2023-01-11T22:21:33.2732809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30640
2023-01-11T22:21:33.2733182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2733362Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2733722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2733916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2734281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2734457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2734831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2735020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2735269Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2735517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2735903Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2736302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2736544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2736774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2737012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.2737253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.2737652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2738049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2738274Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:21:33.2738513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:21:33.2738908Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:21:33.2739300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:21:33.2739540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T22:21:33.2739777Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T22:21:33.2740229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2023-01-11T22:21:33.2740636Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2023-01-11T22:21:33.2740817Z ok (4.312s)
2023-01-11T22:21:33.2740837Z 
2023-01-11T22:21:33.2741087Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2741198Z Ran 1 test in 4.313s
2023-01-11T22:21:33.2741218Z 
2023-01-11T22:21:33.2741310Z OK
2023-01-11T22:21:33.2741330Z 
2023-01-11T22:21:33.2741455Z Generating XML reports...
2023-01-11T22:21:33.2741910Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215630.xml
2023-01-11T22:21:33.2742283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2742457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2742842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2743016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2743054Z 
2023-01-11T22:21:33.2743145Z Running tests...
2023-01-11T22:21:33.2743408Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2743722Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2743982Z   test_all_gather_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports all_gather_v (0.002s)
2023-01-11T22:21:33.2744002Z 
2023-01-11T22:21:33.2744258Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2744368Z Ran 1 test in 0.002s
2023-01-11T22:21:33.2744387Z 
2023-01-11T22:21:33.2744491Z OK (skipped=1)
2023-01-11T22:21:33.2744510Z 
2023-01-11T22:21:33.2744632Z Generating XML reports...
2023-01-11T22:21:33.2745067Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215637.xml
2023-01-11T22:21:33.2745439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2745618Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2745996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2746185Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2746205Z 
2023-01-11T22:21:33.2746311Z Running tests...
2023-01-11T22:21:33.2746572Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2746883Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2747166Z   test_all_reduce_coalesced_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2747373Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30805
2023-01-11T22:21:33.2747590Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30806
2023-01-11T22:21:33.2747963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2748138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2748518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2748705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2749067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2749240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2749648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2749838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2750129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2750373Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2750774Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2751170Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2751398Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2751628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2751870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.2752095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.2752494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2752886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2753221Z STAGE:2023-01-11 21:56:43 30806:30806 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2753543Z STAGE:2023-01-11 21:56:43 30805:30805 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2754291Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2754404Z   warnings.warn(
2023-01-11T22:21:33.2755144Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2755257Z   warnings.warn(
2023-01-11T22:21:33.2755803Z STAGE:2023-01-11 21:56:43 30806:30806 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:56:43 30805:30805 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2755824Z 
2023-01-11T22:21:33.2756391Z STAGE:2023-01-11 21:56:43 30806:30806 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:56:43 30805:30805 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2756412Z 
2023-01-11T22:21:33.2756742Z STAGE:2023-01-11 21:56:43 30806:30806 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2757054Z STAGE:2023-01-11 21:56:43 30805:30805 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2757603Z STAGE:2023-01-11 21:56:43 30806:30806 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:56:43 30805:30805 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2757622Z 
2023-01-11T22:21:33.2757971Z STAGE:2023-01-11 21:56:43 30806:30806 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2758320Z STAGE:2023-01-11 21:56:43 30805:30805 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2758422Z ok (4.259s)
2023-01-11T22:21:33.2758441Z 
2023-01-11T22:21:33.2758703Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2758814Z Ran 1 test in 4.259s
2023-01-11T22:21:33.2758833Z 
2023-01-11T22:21:33.2758976Z OK
2023-01-11T22:21:33.2758998Z 
2023-01-11T22:21:33.2759124Z Generating XML reports...
2023-01-11T22:21:33.2759567Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215639.xml
2023-01-11T22:21:33.2760001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2760175Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2760555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2760747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2760766Z 
2023-01-11T22:21:33.2760872Z Running tests...
2023-01-11T22:21:33.2761133Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2761451Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2761736Z   test_all_reduce_coalesced_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2761943Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 30924
2023-01-11T22:21:33.2762160Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 30925
2023-01-11T22:21:33.2762533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2762707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2763085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2763277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2763647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2763820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2764174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2764603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2764849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2765094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2765502Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2765898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2766135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2766369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2766610Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.2766841Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.2767239Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2767633Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2767967Z STAGE:2023-01-11 21:56:50 30924:30924 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2768291Z STAGE:2023-01-11 21:56:50 30925:30925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2769107Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2769270Z   warnings.warn(
2023-01-11T22:21:33.2770019Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2770128Z   warnings.warn(
2023-01-11T22:21:33.2770678Z STAGE:2023-01-11 21:56:50 30924:30924 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:56:50 30925:30925 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2770698Z 
2023-01-11T22:21:33.2771270Z STAGE:2023-01-11 21:56:50 30924:30924 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:56:50 30925:30925 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2771291Z 
2023-01-11T22:21:33.2771617Z STAGE:2023-01-11 21:56:50 30924:30924 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2771921Z STAGE:2023-01-11 21:56:50 30925:30925 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2772254Z STAGE:2023-01-11 21:56:50 30925:30925 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2772589Z STAGE:2023-01-11 21:56:50 30924:30924 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2772935Z STAGE:2023-01-11 21:56:50 30925:30925 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2773280Z STAGE:2023-01-11 21:56:50 30924:30924 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2773380Z ok (4.211s)
2023-01-11T22:21:33.2773403Z 
2023-01-11T22:21:33.2773667Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2773778Z Ran 1 test in 4.211s
2023-01-11T22:21:33.2773801Z 
2023-01-11T22:21:33.2773875Z OK
2023-01-11T22:21:33.2773910Z 
2023-01-11T22:21:33.2774017Z Generating XML reports...
2023-01-11T22:21:33.2774475Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215646.xml
2023-01-11T22:21:33.2774849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2775025Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2775407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2775598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2775618Z 
2023-01-11T22:21:33.2775724Z Running tests...
2023-01-11T22:21:33.2775988Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2776289Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2776585Z   test_all_reduce_coalesced_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2776806Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31043
2023-01-11T22:21:33.2777024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31044
2023-01-11T22:21:33.2777395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2777567Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2777946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2778184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2778540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2778756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2779129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2779316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2779564Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2779807Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2780209Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2780610Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2780842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2781058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2781302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.2781544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.2781940Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2782333Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2782665Z STAGE:2023-01-11 21:56:57 31044:31044 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2782989Z STAGE:2023-01-11 21:56:57 31043:31043 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2783732Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2783848Z   warnings.warn(
2023-01-11T22:21:33.2784587Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2784682Z   warnings.warn(
2023-01-11T22:21:33.2785234Z STAGE:2023-01-11 21:56:57 31043:31043 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:56:57 31044:31044 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2785254Z 
2023-01-11T22:21:33.2785828Z STAGE:2023-01-11 21:56:57 31044:31044 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:56:57 31043:31043 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2785850Z 
2023-01-11T22:21:33.2786177Z STAGE:2023-01-11 21:56:57 31044:31044 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2786497Z STAGE:2023-01-11 21:56:57 31043:31043 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2786831Z STAGE:2023-01-11 21:56:57 31044:31044 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2787164Z STAGE:2023-01-11 21:56:57 31043:31043 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2787510Z STAGE:2023-01-11 21:56:57 31044:31044 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2787904Z STAGE:2023-01-11 21:56:57 31043:31043 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2788012Z ok (4.354s)
2023-01-11T22:21:33.2788071Z 
2023-01-11T22:21:33.2788339Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2788436Z Ran 1 test in 4.354s
2023-01-11T22:21:33.2788455Z 
2023-01-11T22:21:33.2788547Z OK
2023-01-11T22:21:33.2788565Z 
2023-01-11T22:21:33.2788689Z Generating XML reports...
2023-01-11T22:21:33.2789142Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215653.xml
2023-01-11T22:21:33.2789512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2789688Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2790074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2790265Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2790285Z 
2023-01-11T22:21:33.2790375Z Running tests...
2023-01-11T22:21:33.2790640Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2790953Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2791238Z   test_all_reduce_coalesced_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2791457Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31162
2023-01-11T22:21:33.2791677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31163
2023-01-11T22:21:33.2792046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2792220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2792601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2792777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2793143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2793314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2793688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2793873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2794119Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2794364Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2794769Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2795151Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2795386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2795612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2795850Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.2796092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.2796487Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2796930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2797275Z STAGE:2023-01-11 21:57:04 31162:31162 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2797648Z STAGE:2023-01-11 21:57:04 31163:31163 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2798398Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2798495Z   warnings.warn(
2023-01-11T22:21:33.2799230Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2799344Z   warnings.warn(
2023-01-11T22:21:33.2799888Z STAGE:2023-01-11 21:57:04 31163:31163 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:57:04 31162:31162 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2799912Z 
2023-01-11T22:21:33.2800482Z STAGE:2023-01-11 21:57:04 31163:31163 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:57:04 31162:31162 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2800502Z 
2023-01-11T22:21:33.2800823Z STAGE:2023-01-11 21:57:04 31163:31163 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2801140Z STAGE:2023-01-11 21:57:04 31162:31162 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2801472Z STAGE:2023-01-11 21:57:04 31163:31163 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2801806Z STAGE:2023-01-11 21:57:04 31162:31162 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2802150Z STAGE:2023-01-11 21:57:04 31163:31163 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2802499Z STAGE:2023-01-11 21:57:04 31162:31162 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2802583Z ok (4.244s)
2023-01-11T22:21:33.2802602Z 
2023-01-11T22:21:33.2802863Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2802973Z Ran 1 test in 4.244s
2023-01-11T22:21:33.2802992Z 
2023-01-11T22:21:33.2803085Z OK
2023-01-11T22:21:33.2803104Z 
2023-01-11T22:21:33.2803226Z Generating XML reports...
2023-01-11T22:21:33.2803678Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215700.xml
2023-01-11T22:21:33.2804047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2804398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2804781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2804977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2804997Z 
2023-01-11T22:21:33.2805103Z Running tests...
2023-01-11T22:21:33.2805363Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2805676Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2805956Z   test_all_reduce_coalesced_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2806177Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31281
2023-01-11T22:21:33.2806394Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31282
2023-01-11T22:21:33.2806838Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2807007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2807446Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2807636Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2808004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2808176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2808547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2808736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2808985Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2809214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2809616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2810057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2810290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2810518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2810677Z skip: Skipped due to small world size. (4.230s)
2023-01-11T22:21:33.2810696Z 
2023-01-11T22:21:33.2810962Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2811073Z Ran 1 test in 4.230s
2023-01-11T22:21:33.2811093Z 
2023-01-11T22:21:33.2811201Z OK (skipped=1)
2023-01-11T22:21:33.2811223Z 
2023-01-11T22:21:33.2811330Z Generating XML reports...
2023-01-11T22:21:33.2811783Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215706.xml
2023-01-11T22:21:33.2812158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2812334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2812713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2812904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2812924Z 
2023-01-11T22:21:33.2813029Z Running tests...
2023-01-11T22:21:33.2813292Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2813603Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2813868Z   test_all_reduce_coalesced_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2814088Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31390
2023-01-11T22:21:33.2814309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31391
2023-01-11T22:21:33.2814681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2814853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2815233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2815424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2815789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2815997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2816385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2816621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2816868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2817113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2817517Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2817914Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2818143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2818375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2818515Z skip: Skipped due to small world size. (4.224s)
2023-01-11T22:21:33.2818537Z 
2023-01-11T22:21:33.2818803Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2818916Z Ran 1 test in 4.225s
2023-01-11T22:21:33.2818935Z 
2023-01-11T22:21:33.2819042Z OK (skipped=1)
2023-01-11T22:21:33.2819061Z 
2023-01-11T22:21:33.2819183Z Generating XML reports...
2023-01-11T22:21:33.2819636Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215713.xml
2023-01-11T22:21:33.2820005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2820180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2820561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2820736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2820756Z 
2023-01-11T22:21:33.2820865Z Running tests...
2023-01-11T22:21:33.2821225Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2821764Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2822238Z   test_all_reduce_coalesced_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2822594Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31499
2023-01-11T22:21:33.2822960Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31500
2023-01-11T22:21:33.2823425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2823584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2823973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2824166Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2824528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2824701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2825075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2825261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2825509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2825752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2826214Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2826626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2826921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2827151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2827308Z skip: Skipped due to small world size. (4.134s)
2023-01-11T22:21:33.2827329Z 
2023-01-11T22:21:33.2827592Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2827703Z Ran 1 test in 4.134s
2023-01-11T22:21:33.2827722Z 
2023-01-11T22:21:33.2827829Z OK (skipped=1)
2023-01-11T22:21:33.2827847Z 
2023-01-11T22:21:33.2827970Z Generating XML reports...
2023-01-11T22:21:33.2828413Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215720.xml
2023-01-11T22:21:33.2828783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2828961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2829340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2829532Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2829551Z 
2023-01-11T22:21:33.2829658Z Running tests...
2023-01-11T22:21:33.2829920Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2830233Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2830496Z   test_all_reduce_coalesced_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2830722Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31608
2023-01-11T22:21:33.2830938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31609
2023-01-11T22:21:33.2831347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2831522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2831901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2832090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2832452Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2832622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2832987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2833175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2833420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2833665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2834063Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2834461Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2834692Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2834919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2835059Z skip: Skipped due to small world size. (4.238s)
2023-01-11T22:21:33.2835149Z 
2023-01-11T22:21:33.2835404Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2835515Z Ran 1 test in 4.238s
2023-01-11T22:21:33.2835577Z 
2023-01-11T22:21:33.2835686Z OK (skipped=1)
2023-01-11T22:21:33.2835705Z 
2023-01-11T22:21:33.2835828Z Generating XML reports...
2023-01-11T22:21:33.2836283Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215727.xml
2023-01-11T22:21:33.2836652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2836825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2837200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2837372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2837407Z 
2023-01-11T22:21:33.2837501Z Running tests...
2023-01-11T22:21:33.2837763Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2838074Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2838348Z   test_all_reduce_coalesced_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2838565Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31717
2023-01-11T22:21:33.2838782Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31718
2023-01-11T22:21:33.2839156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2839328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2839690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2839883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2840246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2840421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2840791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2840977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2841221Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2841464Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2841849Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2842249Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2842479Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2842710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2843044Z STAGE:2023-01-11 21:57:37 31718:31718 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2843365Z STAGE:2023-01-11 21:57:37 31717:31717 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2844111Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2844429Z   warnings.warn(
2023-01-11T22:21:33.2845274Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2845439Z   warnings.warn(
2023-01-11T22:21:33.2845977Z STAGE:2023-01-11 21:57:37 31717:31717 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:57:37 31718:31718 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2846015Z 
2023-01-11T22:21:33.2846571Z STAGE:2023-01-11 21:57:37 31718:31718 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:57:37 31717:31717 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2846607Z 
2023-01-11T22:21:33.2846913Z STAGE:2023-01-11 21:57:37 31718:31718 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2847241Z STAGE:2023-01-11 21:57:37 31717:31717 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2847782Z STAGE:2023-01-11 21:57:37 31717:31717 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:57:37 31718:31718 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2847805Z 
2023-01-11T22:21:33.2848373Z STAGE:2023-01-11 21:57:37 31718:31718 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:57:37 31717:31717 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2848393Z 
2023-01-11T22:21:33.2848495Z ok (4.237s)
2023-01-11T22:21:33.2848514Z 
2023-01-11T22:21:33.2848775Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2848885Z Ran 1 test in 4.238s
2023-01-11T22:21:33.2848904Z 
2023-01-11T22:21:33.2848995Z OK
2023-01-11T22:21:33.2849014Z 
2023-01-11T22:21:33.2849135Z Generating XML reports...
2023-01-11T22:21:33.2849595Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215734.xml
2023-01-11T22:21:33.2849953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2850134Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2850519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2850711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2850731Z 
2023-01-11T22:21:33.2850837Z Running tests...
2023-01-11T22:21:33.2851096Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2851414Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2851720Z   test_all_reduce_coalesced_max_complex_unsupported (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2851925Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31830
2023-01-11T22:21:33.2852143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31831
2023-01-11T22:21:33.2852516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2852691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2853069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2853258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2853619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2853791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2854213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2854394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2854682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2854926Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2855328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2855723Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2855954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2856184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2856929Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2857046Z   warnings.warn(
2023-01-11T22:21:33.2857783Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2857878Z   warnings.warn(
2023-01-11T22:21:33.2857977Z ok (4.240s)
2023-01-11T22:21:33.2857997Z 
2023-01-11T22:21:33.2858259Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2858371Z Ran 1 test in 4.240s
2023-01-11T22:21:33.2858390Z 
2023-01-11T22:21:33.2858482Z OK
2023-01-11T22:21:33.2858501Z 
2023-01-11T22:21:33.2858627Z Generating XML reports...
2023-01-11T22:21:33.2859082Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215740.xml
2023-01-11T22:21:33.2859456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2859615Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2859992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2860181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2860200Z 
2023-01-11T22:21:33.2860306Z Running tests...
2023-01-11T22:21:33.2860565Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2860875Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2861149Z   test_all_reduce_coalesced_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2861372Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 31939
2023-01-11T22:21:33.2861592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 31940
2023-01-11T22:21:33.2861942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2862116Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2862488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2862682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2863039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2863260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2863639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2863871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2864102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2864347Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2864749Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2865145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2865375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2865606Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2865940Z STAGE:2023-01-11 21:57:51 31940:31940 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2866267Z STAGE:2023-01-11 21:57:51 31939:31939 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2867007Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2867121Z   warnings.warn(
2023-01-11T22:21:33.2867840Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2867954Z   warnings.warn(
2023-01-11T22:21:33.2868290Z STAGE:2023-01-11 21:57:51 31939:31939 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2868616Z STAGE:2023-01-11 21:57:51 31940:31940 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2868969Z STAGE:2023-01-11 21:57:51 31939:31939 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2869314Z STAGE:2023-01-11 21:57:51 31940:31940 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2869639Z STAGE:2023-01-11 21:57:51 31940:31940 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2869959Z STAGE:2023-01-11 21:57:51 31939:31939 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2870291Z STAGE:2023-01-11 21:57:51 31939:31939 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2870610Z STAGE:2023-01-11 21:57:51 31940:31940 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2870957Z STAGE:2023-01-11 21:57:51 31939:31939 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2871302Z STAGE:2023-01-11 21:57:51 31940:31940 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2871404Z ok (4.208s)
2023-01-11T22:21:33.2871423Z 
2023-01-11T22:21:33.2871688Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2871798Z Ran 1 test in 4.208s
2023-01-11T22:21:33.2871818Z 
2023-01-11T22:21:33.2871909Z OK
2023-01-11T22:21:33.2871929Z 
2023-01-11T22:21:33.2872053Z Generating XML reports...
2023-01-11T22:21:33.2872492Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215747.xml
2023-01-11T22:21:33.2872863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2873088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2873479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2873715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2873734Z 
2023-01-11T22:21:33.2873841Z Running tests...
2023-01-11T22:21:33.2874103Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2874414Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2874694Z   test_all_reduce_coalesced_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2874899Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32052
2023-01-11T22:21:33.2875116Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32053
2023-01-11T22:21:33.2875488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2875664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2876048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2876238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2876601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2876773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2877144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2877314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2877561Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2877808Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2878211Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2878612Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2878842Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2879070Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2879406Z STAGE:2023-01-11 21:57:58 32053:32053 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2879712Z STAGE:2023-01-11 21:57:58 32052:32052 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2880461Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2880577Z   warnings.warn(
2023-01-11T22:21:33.2881316Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2881426Z   warnings.warn(
2023-01-11T22:21:33.2881974Z STAGE:2023-01-11 21:57:58 32052:32052 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:57:58 32053:32053 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2881995Z 
2023-01-11T22:21:33.2882614Z STAGE:2023-01-11 21:57:58 32052:32052 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:57:58 32053:32053 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2882636Z 
2023-01-11T22:21:33.2883004Z STAGE:2023-01-11 21:57:58 32052:32052 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2883325Z STAGE:2023-01-11 21:57:58 32053:32053 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2883865Z STAGE:2023-01-11 21:57:58 32053:32053 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:57:58 32052:32052 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2883885Z 
2023-01-11T22:21:33.2884681Z STAGE:2023-01-11 21:57:58 32053:32053 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:57:58 32052:32052 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2884705Z 
2023-01-11T22:21:33.2884811Z ok (4.210s)
2023-01-11T22:21:33.2884830Z 
2023-01-11T22:21:33.2885103Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2885199Z Ran 1 test in 4.210s
2023-01-11T22:21:33.2885234Z 
2023-01-11T22:21:33.2885314Z OK
2023-01-11T22:21:33.2885334Z 
2023-01-11T22:21:33.2885456Z Generating XML reports...
2023-01-11T22:21:33.2885910Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215754.xml
2023-01-11T22:21:33.2886282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2886459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2886841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2887033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2887053Z 
2023-01-11T22:21:33.2887159Z Running tests...
2023-01-11T22:21:33.2887408Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2887724Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2887999Z   test_all_reduce_coalesced_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2888219Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32165
2023-01-11T22:21:33.2888437Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32166
2023-01-11T22:21:33.2888806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2888980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2889358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2889535Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2889905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2890081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2890453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2890639Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2890885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2891130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2891530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2892034Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2892257Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2892549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2892894Z STAGE:2023-01-11 21:58:04 32166:32166 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2893635Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2893750Z   warnings.warn(
2023-01-11T22:21:33.2894080Z STAGE:2023-01-11 21:58:04 32165:32165 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2894822Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1714: UserWarning: torch.distributed.all_reduce_coalesced will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#collective-functions
2023-01-11T22:21:33.2894936Z   warnings.warn(
2023-01-11T22:21:33.2895272Z STAGE:2023-01-11 21:58:04 32166:32166 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2895620Z STAGE:2023-01-11 21:58:04 32166:32166 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2895936Z STAGE:2023-01-11 21:58:04 32165:32165 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2896279Z STAGE:2023-01-11 21:58:04 32165:32165 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2896606Z STAGE:2023-01-11 21:58:04 32166:32166 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2896929Z STAGE:2023-01-11 21:58:04 32165:32165 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2897263Z STAGE:2023-01-11 21:58:04 32166:32166 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2897608Z STAGE:2023-01-11 21:58:04 32166:32166 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2897942Z STAGE:2023-01-11 21:58:04 32165:32165 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2898286Z STAGE:2023-01-11 21:58:04 32165:32165 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2898386Z ok (4.335s)
2023-01-11T22:21:33.2898407Z 
2023-01-11T22:21:33.2898657Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2898768Z Ran 1 test in 4.335s
2023-01-11T22:21:33.2898787Z 
2023-01-11T22:21:33.2898878Z OK
2023-01-11T22:21:33.2898897Z 
2023-01-11T22:21:33.2899020Z Generating XML reports...
2023-01-11T22:21:33.2899476Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215801.xml
2023-01-11T22:21:33.2899848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2900029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2900414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2900590Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2900625Z 
2023-01-11T22:21:33.2900718Z Running tests...
2023-01-11T22:21:33.2900982Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2901297Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2901584Z   test_all_reduce_complex_unsupported_ops (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2901857Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32278
2023-01-11T22:21:33.2902082Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32279
2023-01-11T22:21:33.2902500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2902675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2903037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2903227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2903588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2903761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2904138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2904329Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2904637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2904930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2905317Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2905752Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2906134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2906405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2906543Z ok (4.230s)
2023-01-11T22:21:33.2906563Z 
2023-01-11T22:21:33.2906888Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2907039Z Ran 1 test in 4.230s
2023-01-11T22:21:33.2907060Z 
2023-01-11T22:21:33.2907138Z OK
2023-01-11T22:21:33.2907157Z 
2023-01-11T22:21:33.2907318Z Generating XML reports...
2023-01-11T22:21:33.2907802Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215807.xml
2023-01-11T22:21:33.2908216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2908474Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2908896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2909134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2909155Z 
2023-01-11T22:21:33.2909339Z Running tests...
2023-01-11T22:21:33.2909642Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2909943Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2910301Z   test_all_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2910559Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32387
2023-01-11T22:21:33.2910815Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32388
2023-01-11T22:21:33.2911267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2911492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2911910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2937686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2938182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2938439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2938839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2939028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2939261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2939503Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2939909Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2940310Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2940537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2940772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2941009Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.2941244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.2941646Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2942024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2942353Z STAGE:2023-01-11 21:58:18 32388:32388 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2942677Z STAGE:2023-01-11 21:58:18 32387:32387 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2943015Z STAGE:2023-01-11 21:58:18 32387:32387 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2943342Z STAGE:2023-01-11 21:58:18 32388:32388 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2943683Z STAGE:2023-01-11 21:58:18 32387:32387 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2944027Z STAGE:2023-01-11 21:58:18 32388:32388 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2944348Z STAGE:2023-01-11 21:58:18 32387:32387 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2944667Z STAGE:2023-01-11 21:58:18 32388:32388 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2945200Z STAGE:2023-01-11 21:58:18 32387:32387 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:58:18 32388:32388 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2945241Z 
2023-01-11T22:21:33.2945571Z STAGE:2023-01-11 21:58:18 32387:32387 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2945913Z STAGE:2023-01-11 21:58:18 32388:32388 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2946008Z ok (4.229s)
2023-01-11T22:21:33.2946028Z 
2023-01-11T22:21:33.2946289Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2946397Z Ran 1 test in 4.230s
2023-01-11T22:21:33.2946417Z 
2023-01-11T22:21:33.2946510Z OK
2023-01-11T22:21:33.2946529Z 
2023-01-11T22:21:33.2946653Z Generating XML reports...
2023-01-11T22:21:33.2947107Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215814.xml
2023-01-11T22:21:33.2947515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2947693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2948081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2948318Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2948337Z 
2023-01-11T22:21:33.2948438Z Running tests...
2023-01-11T22:21:33.2948699Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2949009Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2949275Z   test_all_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2949494Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32506
2023-01-11T22:21:33.2949698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32507
2023-01-11T22:21:33.2950067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2950238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2950625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2950809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2951164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2951338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2951709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2951884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2952130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2952378Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2952781Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2953169Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2953402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2953625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2953861Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.2954106Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.2954494Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2954891Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2955225Z STAGE:2023-01-11 21:58:25 32506:32506 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2955542Z STAGE:2023-01-11 21:58:25 32507:32507 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2956091Z STAGE:2023-01-11 21:58:25 32507:32507 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:58:25 32506:32506 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2956111Z 
2023-01-11T22:21:33.2956726Z STAGE:2023-01-11 21:58:25 32507:32507 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:58:25 32506:32506 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2956748Z 
2023-01-11T22:21:33.2957074Z STAGE:2023-01-11 21:58:25 32507:32507 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2957441Z STAGE:2023-01-11 21:58:25 32506:32506 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2957768Z STAGE:2023-01-11 21:58:25 32507:32507 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2958095Z STAGE:2023-01-11 21:58:25 32506:32506 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2958440Z STAGE:2023-01-11 21:58:25 32507:32507 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2958768Z STAGE:2023-01-11 21:58:25 32506:32506 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2958868Z ok (4.335s)
2023-01-11T22:21:33.2958887Z 
2023-01-11T22:21:33.2959143Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2959258Z Ran 1 test in 4.336s
2023-01-11T22:21:33.2959278Z 
2023-01-11T22:21:33.2959363Z OK
2023-01-11T22:21:33.2959382Z 
2023-01-11T22:21:33.2959500Z Generating XML reports...
2023-01-11T22:21:33.2959958Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215821.xml
2023-01-11T22:21:33.2960328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2960490Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2960870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2961062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2961081Z 
2023-01-11T22:21:33.2961183Z Running tests...
2023-01-11T22:21:33.2961437Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2961757Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2962032Z   test_all_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2962250Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32625
2023-01-11T22:21:33.2962467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32626
2023-01-11T22:21:33.2962823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2962994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2963373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2963564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2963928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2964094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2964703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2964887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2965117Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2965355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2965761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2966155Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2966466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2966705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2966995Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.2967229Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.2967635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2968031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2968351Z STAGE:2023-01-11 21:58:32 32625:32625 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2968674Z STAGE:2023-01-11 21:58:32 32626:32626 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2969218Z STAGE:2023-01-11 21:58:32 32626:32626 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:58:32 32625:32625 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2969242Z 
2023-01-11T22:21:33.2969818Z STAGE:2023-01-11 21:58:32 32625:32625 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:58:32 32626:32626 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2969838Z 
2023-01-11T22:21:33.2970362Z STAGE:2023-01-11 21:58:32 32626:32626 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:58:32 32625:32625 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2970382Z 
2023-01-11T22:21:33.2970709Z STAGE:2023-01-11 21:58:32 32626:32626 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2971040Z STAGE:2023-01-11 21:58:32 32625:32625 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2971380Z STAGE:2023-01-11 21:58:32 32626:32626 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2971722Z STAGE:2023-01-11 21:58:32 32625:32625 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2971825Z ok (4.258s)
2023-01-11T22:21:33.2971844Z 
2023-01-11T22:21:33.2972101Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2972195Z Ran 1 test in 4.258s
2023-01-11T22:21:33.2972214Z 
2023-01-11T22:21:33.2972300Z OK
2023-01-11T22:21:33.2972319Z 
2023-01-11T22:21:33.2972442Z Generating XML reports...
2023-01-11T22:21:33.2972897Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215828.xml
2023-01-11T22:21:33.2973262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2973438Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2973822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2974011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2974031Z 
2023-01-11T22:21:33.2974132Z Running tests...
2023-01-11T22:21:33.2974381Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2974698Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2974961Z   test_all_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2975178Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32744
2023-01-11T22:21:33.2975395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32745
2023-01-11T22:21:33.2975820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2975995Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2976422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2976595Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2976958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2977126Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2977513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2977702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2977947Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2978194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2978587Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2978986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2979203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2979424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2979656Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.2979900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.2980297Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2980682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.2981021Z STAGE:2023-01-11 21:58:38 32745:32745 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2981339Z STAGE:2023-01-11 21:58:38 32744:32744 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2981667Z STAGE:2023-01-11 21:58:38 32744:32744 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2981975Z STAGE:2023-01-11 21:58:38 32745:32745 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2982322Z STAGE:2023-01-11 21:58:38 32744:32744 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2982663Z STAGE:2023-01-11 21:58:38 32745:32745 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2982987Z STAGE:2023-01-11 21:58:38 32744:32744 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2983310Z STAGE:2023-01-11 21:58:38 32745:32745 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.2983642Z STAGE:2023-01-11 21:58:38 32744:32744 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2983966Z STAGE:2023-01-11 21:58:38 32745:32745 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.2984311Z STAGE:2023-01-11 21:58:38 32744:32744 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2984652Z STAGE:2023-01-11 21:58:38 32745:32745 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.2984738Z ok (4.210s)
2023-01-11T22:21:33.2984758Z 
2023-01-11T22:21:33.2985017Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2985129Z Ran 1 test in 4.211s
2023-01-11T22:21:33.2985217Z 
2023-01-11T22:21:33.2985310Z OK
2023-01-11T22:21:33.2985329Z 
2023-01-11T22:21:33.2985445Z Generating XML reports...
2023-01-11T22:21:33.2985901Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215835.xml
2023-01-11T22:21:33.2986317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2986489Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2986856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2987045Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2987064Z 
2023-01-11T22:21:33.2987165Z Running tests...
2023-01-11T22:21:33.2987424Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2987745Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2988004Z   test_all_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2988221Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32863
2023-01-11T22:21:33.2988441Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32864
2023-01-11T22:21:33.2988800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2988972Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2989349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2989537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2989903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2990069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2990444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2990631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2990870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2991097Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2991499Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2991896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2992180Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.2992406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.2992564Z skip: Skipped due to small world size. (4.126s)
2023-01-11T22:21:33.2992587Z 
2023-01-11T22:21:33.2992850Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2992955Z Ran 1 test in 4.126s
2023-01-11T22:21:33.2992975Z 
2023-01-11T22:21:33.2993082Z OK (skipped=1)
2023-01-11T22:21:33.2993101Z 
2023-01-11T22:21:33.2993208Z Generating XML reports...
2023-01-11T22:21:33.2993660Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215841.xml
2023-01-11T22:21:33.2994025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2994202Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2994631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2994823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2994881Z 
2023-01-11T22:21:33.2994985Z Running tests...
2023-01-11T22:21:33.2995243Z ----------------------------------------------------------------------
2023-01-11T22:21:33.2995539Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.2995799Z   test_all_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.2996017Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 32972
2023-01-11T22:21:33.2996235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 32973
2023-01-11T22:21:33.2996599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2996772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2997146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2997335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2997700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.2997854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.2998231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.2998412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.2998651Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.2998896Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.2999296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2999689Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.2999923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3000146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3000288Z skip: Skipped due to small world size. (4.160s)
2023-01-11T22:21:33.3000307Z 
2023-01-11T22:21:33.3000564Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3000671Z Ran 1 test in 4.160s
2023-01-11T22:21:33.3000691Z 
2023-01-11T22:21:33.3000790Z OK (skipped=1)
2023-01-11T22:21:33.3000809Z 
2023-01-11T22:21:33.3000926Z Generating XML reports...
2023-01-11T22:21:33.3001383Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215848.xml
2023-01-11T22:21:33.3001749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3001918Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3002283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3002474Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3002493Z 
2023-01-11T22:21:33.3002600Z Running tests...
2023-01-11T22:21:33.3002860Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3003168Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3003439Z   test_all_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3003712Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33081
2023-01-11T22:21:33.3003931Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33082
2023-01-11T22:21:33.3004622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3004790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3005175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3005360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3005715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3005890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3006265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3006447Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3006693Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3006922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3007328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3007721Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3007945Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3008174Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3008330Z skip: Skipped due to small world size. (4.236s)
2023-01-11T22:21:33.3008350Z 
2023-01-11T22:21:33.3008608Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3008722Z Ran 1 test in 4.236s
2023-01-11T22:21:33.3008744Z 
2023-01-11T22:21:33.3008854Z OK (skipped=1)
2023-01-11T22:21:33.3008873Z 
2023-01-11T22:21:33.3008977Z Generating XML reports...
2023-01-11T22:21:33.3009426Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215855.xml
2023-01-11T22:21:33.3009790Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3009959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3010371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3010566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3010588Z 
2023-01-11T22:21:33.3010690Z Running tests...
2023-01-11T22:21:33.3010946Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3011263Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3011510Z   test_all_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3011726Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33190
2023-01-11T22:21:33.3011939Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33191
2023-01-11T22:21:33.3012309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3012478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3012852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3013124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3013500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3013713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3014083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3014271Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3014513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3014752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3015153Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3015547Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3015774Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3016006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3016146Z skip: Skipped due to small world size. (4.203s)
2023-01-11T22:21:33.3016165Z 
2023-01-11T22:21:33.3016423Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3016527Z Ran 1 test in 4.203s
2023-01-11T22:21:33.3016546Z 
2023-01-11T22:21:33.3016651Z OK (skipped=1)
2023-01-11T22:21:33.3016670Z 
2023-01-11T22:21:33.3016788Z Generating XML reports...
2023-01-11T22:21:33.3017234Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215901.xml
2023-01-11T22:21:33.3017601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3017771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3018153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3018332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3018351Z 
2023-01-11T22:21:33.3018460Z Running tests...
2023-01-11T22:21:33.3018717Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3019022Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3019269Z   test_all_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3019480Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33299
2023-01-11T22:21:33.3019702Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33300
2023-01-11T22:21:33.3020068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3020228Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3020607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3020802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3021164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3021328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3021711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3021902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3022193Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3022433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3022867Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3023264Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3023491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3023713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3024049Z STAGE:2023-01-11 21:59:12 33300:33300 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3024371Z STAGE:2023-01-11 21:59:12 33299:33299 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3024916Z STAGE:2023-01-11 21:59:12 33299:33299 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:59:12 33300:33300 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3024940Z 
2023-01-11T22:21:33.3025512Z STAGE:2023-01-11 21:59:12 33299:33299 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:59:12 33300:33300 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3025532Z 
2023-01-11T22:21:33.3025855Z STAGE:2023-01-11 21:59:12 33299:33299 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3026171Z STAGE:2023-01-11 21:59:12 33300:33300 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3026715Z STAGE:2023-01-11 21:59:12 33300:33300 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:59:12 33299:33299 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3026739Z 
2023-01-11T22:21:33.3027069Z STAGE:2023-01-11 21:59:12 33299:33299 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3027412Z STAGE:2023-01-11 21:59:12 33300:33300 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3027508Z ok (4.242s)
2023-01-11T22:21:33.3027527Z 
2023-01-11T22:21:33.3027790Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3027894Z Ran 1 test in 4.242s
2023-01-11T22:21:33.3027913Z 
2023-01-11T22:21:33.3027997Z OK
2023-01-11T22:21:33.3028017Z 
2023-01-11T22:21:33.3028139Z Generating XML reports...
2023-01-11T22:21:33.3028592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215908.xml
2023-01-11T22:21:33.3028959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3029122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3029494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3029680Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3029699Z 
2023-01-11T22:21:33.3029808Z Running tests...
2023-01-11T22:21:33.3030059Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3030369Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3030622Z   test_all_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3030840Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33412
2023-01-11T22:21:33.3031039Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33413
2023-01-11T22:21:33.3031451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3031624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3032001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3032243Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3032603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3032770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3033137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3033319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3033551Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3033949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3034185Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3034583Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3034808Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3035030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3035368Z STAGE:2023-01-11 21:59:19 33412:33412 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3035688Z STAGE:2023-01-11 21:59:19 33413:33413 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3036229Z STAGE:2023-01-11 21:59:19 33413:33413 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:59:19 33412:33412 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3036251Z 
2023-01-11T22:21:33.3036822Z STAGE:2023-01-11 21:59:19 33413:33413 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:59:19 33412:33412 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3036843Z 
2023-01-11T22:21:33.3037154Z STAGE:2023-01-11 21:59:19 33413:33413 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3037467Z STAGE:2023-01-11 21:59:19 33412:33412 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3037795Z STAGE:2023-01-11 21:59:19 33413:33413 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3038118Z STAGE:2023-01-11 21:59:19 33412:33412 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3038466Z STAGE:2023-01-11 21:59:19 33413:33413 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3038806Z STAGE:2023-01-11 21:59:19 33412:33412 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3038904Z ok (4.197s)
2023-01-11T22:21:33.3038923Z 
2023-01-11T22:21:33.3039188Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3039293Z Ran 1 test in 4.197s
2023-01-11T22:21:33.3039313Z 
2023-01-11T22:21:33.3039386Z OK
2023-01-11T22:21:33.3039405Z 
2023-01-11T22:21:33.3039521Z Generating XML reports...
2023-01-11T22:21:33.3039979Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215915.xml
2023-01-11T22:21:33.3040349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3040520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3040950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3041142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3041201Z 
2023-01-11T22:21:33.3041312Z Running tests...
2023-01-11T22:21:33.3041559Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3041872Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3042127Z   test_all_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3042349Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33525
2023-01-11T22:21:33.3042560Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33526
2023-01-11T22:21:33.3042925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3043104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3043479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3043669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3044019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3044454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3044850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3045042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3045285Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3045529Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3045933Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3046325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3046549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3046761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3047099Z STAGE:2023-01-11 21:59:27 33526:33526 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3047869Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1582: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T22:21:33.3047977Z   warnings.warn(
2023-01-11T22:21:33.3048314Z STAGE:2023-01-11 21:59:27 33525:33525 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3049079Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1582: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T22:21:33.3049185Z   warnings.warn(
2023-01-11T22:21:33.3049522Z STAGE:2023-01-11 21:59:27 33525:33525 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3049852Z STAGE:2023-01-11 21:59:27 33526:33526 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3050192Z STAGE:2023-01-11 21:59:27 33525:33525 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3050618Z STAGE:2023-01-11 21:59:27 33526:33526 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3050962Z STAGE:2023-01-11 21:59:27 33525:33525 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3051337Z STAGE:2023-01-11 21:59:27 33526:33526 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3051662Z STAGE:2023-01-11 21:59:27 33525:33525 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3051990Z STAGE:2023-01-11 21:59:27 33526:33526 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3052332Z STAGE:2023-01-11 21:59:27 33525:33525 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3052666Z STAGE:2023-01-11 21:59:27 33526:33526 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3052767Z ok (5.424s)
2023-01-11T22:21:33.3052787Z 
2023-01-11T22:21:33.3053035Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3053143Z Ran 1 test in 5.424s
2023-01-11T22:21:33.3053163Z 
2023-01-11T22:21:33.3053247Z OK
2023-01-11T22:21:33.3053266Z 
2023-01-11T22:21:33.3053383Z Generating XML reports...
2023-01-11T22:21:33.3053839Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215922.xml
2023-01-11T22:21:33.3054209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3054383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3054756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3054952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3054971Z 
2023-01-11T22:21:33.3055061Z Running tests...
2023-01-11T22:21:33.3055316Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3055627Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3055909Z   test_all_reduce_multigpu_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3056127Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33640
2023-01-11T22:21:33.3056338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33641
2023-01-11T22:21:33.3056709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3056879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3057244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3057431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3057799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3057966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3058337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3058520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3058760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3059005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3059399Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3059780Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3060061Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3060297Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3060677Z STAGE:2023-01-11 21:59:34 33640:33640 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3061450Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1582: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T22:21:33.3061563Z   warnings.warn(
2023-01-11T22:21:33.3061894Z STAGE:2023-01-11 21:59:34 33641:33641 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3062661Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1582: UserWarning: torch.distributed.all_reduce_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T22:21:33.3062772Z   warnings.warn(
2023-01-11T22:21:33.3063324Z STAGE:2023-01-11 21:59:35 33641:33641 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:59:35 33640:33640 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3063345Z 
2023-01-11T22:21:33.3063917Z STAGE:2023-01-11 21:59:35 33641:33641 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:59:35 33640:33640 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3063939Z 
2023-01-11T22:21:33.3064268Z STAGE:2023-01-11 21:59:35 33641:33641 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3064572Z STAGE:2023-01-11 21:59:35 33640:33640 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3064907Z STAGE:2023-01-11 21:59:35 33640:33640 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3065242Z STAGE:2023-01-11 21:59:35 33641:33641 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3065591Z STAGE:2023-01-11 21:59:35 33640:33640 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3065937Z STAGE:2023-01-11 21:59:35 33641:33641 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3066040Z ok (5.307s)
2023-01-11T22:21:33.3066059Z 
2023-01-11T22:21:33.3066323Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3066433Z Ran 1 test in 5.307s
2023-01-11T22:21:33.3066453Z 
2023-01-11T22:21:33.3066527Z OK
2023-01-11T22:21:33.3066546Z 
2023-01-11T22:21:33.3066669Z Generating XML reports...
2023-01-11T22:21:33.3067132Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215930.xml
2023-01-11T22:21:33.3067510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3067687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3068075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3068270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3068289Z 
2023-01-11T22:21:33.3068398Z Running tests...
2023-01-11T22:21:33.3068664Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3068965Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3069230Z   test_all_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3069450Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33755
2023-01-11T22:21:33.3069721Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33756
2023-01-11T22:21:33.3070102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3070323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3070706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3070899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3071247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3071420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3071795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3071987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3072238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3072487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3072893Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3073295Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3073526Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3073738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3074071Z STAGE:2023-01-11 21:59:41 33756:33756 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3074399Z STAGE:2023-01-11 21:59:41 33755:33755 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3074949Z STAGE:2023-01-11 21:59:41 33756:33756 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:59:41 33755:33755 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3074973Z 
2023-01-11T22:21:33.3075543Z STAGE:2023-01-11 21:59:41 33756:33756 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:59:41 33755:33755 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3075563Z 
2023-01-11T22:21:33.3075892Z STAGE:2023-01-11 21:59:41 33756:33756 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3076211Z STAGE:2023-01-11 21:59:41 33755:33755 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3076546Z STAGE:2023-01-11 21:59:41 33756:33756 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3076878Z STAGE:2023-01-11 21:59:41 33755:33755 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3077223Z STAGE:2023-01-11 21:59:41 33756:33756 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3077573Z STAGE:2023-01-11 21:59:41 33755:33755 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3077657Z ok (4.130s)
2023-01-11T22:21:33.3077676Z 
2023-01-11T22:21:33.3077940Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3078050Z Ran 1 test in 4.130s
2023-01-11T22:21:33.3078070Z 
2023-01-11T22:21:33.3078161Z OK
2023-01-11T22:21:33.3078180Z 
2023-01-11T22:21:33.3078303Z Generating XML reports...
2023-01-11T22:21:33.3078761Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215937.xml
2023-01-11T22:21:33.3079183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3079367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3079738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3079978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3079997Z 
2023-01-11T22:21:33.3080106Z Running tests...
2023-01-11T22:21:33.3080366Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3080683Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3080952Z   test_all_reduce_result_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3081171Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33868
2023-01-11T22:21:33.3081389Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33869
2023-01-11T22:21:33.3081750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3081927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3082312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3082503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3082865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3083039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3083418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3083606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3083857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3084086Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3084735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3085141Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3085375Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3085604Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3085707Z ok (5.113s)
2023-01-11T22:21:33.3085727Z 
2023-01-11T22:21:33.3085993Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3086103Z Ran 1 test in 5.113s
2023-01-11T22:21:33.3086122Z 
2023-01-11T22:21:33.3086214Z OK
2023-01-11T22:21:33.3086238Z 
2023-01-11T22:21:33.3086346Z Generating XML reports...
2023-01-11T22:21:33.3086801Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215944.xml
2023-01-11T22:21:33.3087180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3087357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3087739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3087931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3087950Z 
2023-01-11T22:21:33.3088058Z Running tests...
2023-01-11T22:21:33.3088318Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3088689Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3088956Z   test_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3089174Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 33979
2023-01-11T22:21:33.3089448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 33980
2023-01-11T22:21:33.3089826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3090003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3090385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3090576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3090941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3091102Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3091482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3091676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3091924Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3092168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3092571Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3092971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3093202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3093417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3093759Z STAGE:2023-01-11 21:59:56 33980:33980 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3094086Z STAGE:2023-01-11 21:59:56 33979:33979 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3094637Z STAGE:2023-01-11 21:59:56 33979:33979 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:59:56 33980:33980 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3094657Z 
2023-01-11T22:21:33.3095231Z STAGE:2023-01-11 21:59:56 33980:33980 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:59:56 33979:33979 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3095251Z 
2023-01-11T22:21:33.3095783Z STAGE:2023-01-11 21:59:56 33979:33979 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 21:59:56 33980:33980 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3095803Z 
2023-01-11T22:21:33.3096349Z STAGE:2023-01-11 21:59:56 33980:33980 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 21:59:56 33979:33979 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3096373Z 
2023-01-11T22:21:33.3096943Z STAGE:2023-01-11 21:59:56 33979:33979 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 21:59:56 33980:33980 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3096963Z 
2023-01-11T22:21:33.3097065Z ok (4.230s)
2023-01-11T22:21:33.3097084Z 
2023-01-11T22:21:33.3097349Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3097463Z Ran 1 test in 4.230s
2023-01-11T22:21:33.3097482Z 
2023-01-11T22:21:33.3097574Z OK
2023-01-11T22:21:33.3097593Z 
2023-01-11T22:21:33.3097715Z Generating XML reports...
2023-01-11T22:21:33.3098204Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215952.xml
2023-01-11T22:21:33.3098592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3098816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3099204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3099399Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3099419Z 
2023-01-11T22:21:33.3099527Z Running tests...
2023-01-11T22:21:33.3099791Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3100108Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3100379Z   test_all_reduce_sum_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3100584Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34092
2023-01-11T22:21:33.3100805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34093
2023-01-11T22:21:33.3101183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3101361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3101742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3101933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3102301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3102476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3102835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3103025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3103276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3103520Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3103925Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3104325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3104557Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3104787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3105124Z STAGE:2023-01-11 22:00:02 34093:34093 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3105429Z STAGE:2023-01-11 22:00:02 34092:34092 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3105981Z STAGE:2023-01-11 22:00:02 34092:34092 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:00:02 34093:34093 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3106002Z 
2023-01-11T22:21:33.3106572Z STAGE:2023-01-11 22:00:02 34093:34093 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:00:02 34092:34092 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3106593Z 
2023-01-11T22:21:33.3107126Z STAGE:2023-01-11 22:00:02 34092:34092 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 22:00:02 34093:34093 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3107195Z 
2023-01-11T22:21:33.3107540Z STAGE:2023-01-11 22:00:02 34092:34092 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3107867Z STAGE:2023-01-11 22:00:02 34093:34093 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3108263Z STAGE:2023-01-11 22:00:02 34092:34092 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3108613Z STAGE:2023-01-11 22:00:02 34093:34093 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3108714Z ok (4.223s)
2023-01-11T22:21:33.3108734Z 
2023-01-11T22:21:33.3108996Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3109109Z Ran 1 test in 4.223s
2023-01-11T22:21:33.3109129Z 
2023-01-11T22:21:33.3109203Z OK
2023-01-11T22:21:33.3109221Z 
2023-01-11T22:21:33.3109344Z Generating XML reports...
2023-01-11T22:21:33.3109804Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215959.xml
2023-01-11T22:21:33.3110224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3110410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3110794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3110987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3111007Z 
2023-01-11T22:21:33.3111119Z Running tests...
2023-01-11T22:21:33.3111381Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3111679Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3111949Z   test_all_reduce_sum_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3112175Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34205
2023-01-11T22:21:33.3112395Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34206
2023-01-11T22:21:33.3112773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3112948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3113331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3113525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3113870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3114045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3114430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3114623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3114872Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3115120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3115525Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3115921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3116153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3116366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3116765Z STAGE:2023-01-11 22:00:09 34206:34206 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3117109Z STAGE:2023-01-11 22:00:09 34205:34205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3117444Z STAGE:2023-01-11 22:00:09 34205:34205 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3117828Z STAGE:2023-01-11 22:00:09 34206:34206 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3118177Z STAGE:2023-01-11 22:00:09 34205:34205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3118527Z STAGE:2023-01-11 22:00:09 34206:34206 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3118853Z STAGE:2023-01-11 22:00:09 34205:34205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3119176Z STAGE:2023-01-11 22:00:09 34206:34206 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3119495Z STAGE:2023-01-11 22:00:09 34206:34206 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3119826Z STAGE:2023-01-11 22:00:09 34205:34205 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3120404Z STAGE:2023-01-11 22:00:09 34206:34206 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:00:09 34205:34205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3120424Z 
2023-01-11T22:21:33.3120526Z ok (4.249s)
2023-01-11T22:21:33.3120545Z 
2023-01-11T22:21:33.3120809Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3120922Z Ran 1 test in 4.249s
2023-01-11T22:21:33.3120941Z 
2023-01-11T22:21:33.3121031Z OK
2023-01-11T22:21:33.3121051Z 
2023-01-11T22:21:33.3121174Z Generating XML reports...
2023-01-11T22:21:33.3121630Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220005.xml
2023-01-11T22:21:33.3121991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3122172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3122559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3122752Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3122772Z 
2023-01-11T22:21:33.3122879Z Running tests...
2023-01-11T22:21:33.3123141Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3123457Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3123720Z   test_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3123940Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34318
2023-01-11T22:21:33.3124143Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34319
2023-01-11T22:21:33.3124757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3124939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3125324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3125520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3125890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3126062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3126435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3126682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3126940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3127186Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3127652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3128046Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3128279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3128509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3128848Z STAGE:2023-01-11 22:00:17 34319:34319 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3129176Z STAGE:2023-01-11 22:00:17 34318:34318 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3129494Z STAGE:2023-01-11 22:00:17 34318:34318 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3129824Z STAGE:2023-01-11 22:00:17 34319:34319 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3130172Z STAGE:2023-01-11 22:00:17 34318:34318 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3130522Z STAGE:2023-01-11 22:00:17 34319:34319 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3130849Z STAGE:2023-01-11 22:00:17 34319:34319 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3131172Z STAGE:2023-01-11 22:00:17 34318:34318 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3131720Z STAGE:2023-01-11 22:00:18 34319:34319 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:00:18 34318:34318 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3131740Z 
2023-01-11T22:21:33.3132310Z STAGE:2023-01-11 22:00:18 34318:34318 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:00:18 34319:34319 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3132334Z 
2023-01-11T22:21:33.3132661Z STAGE:2023-01-11 22:00:18 34319:34319 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3132982Z STAGE:2023-01-11 22:00:18 34318:34318 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3133528Z STAGE:2023-01-11 22:00:18 34319:34319 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:00:18 34318:34318 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3133549Z 
2023-01-11T22:21:33.3133876Z STAGE:2023-01-11 22:00:18 34318:34318 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3134229Z STAGE:2023-01-11 22:00:18 34319:34319 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3134333Z ok (6.114s)
2023-01-11T22:21:33.3134356Z 
2023-01-11T22:21:33.3134620Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3134735Z Ran 1 test in 6.114s
2023-01-11T22:21:33.3134754Z 
2023-01-11T22:21:33.3134844Z OK
2023-01-11T22:21:33.3134863Z 
2023-01-11T22:21:33.3134986Z Generating XML reports...
2023-01-11T22:21:33.3135443Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220012.xml
2023-01-11T22:21:33.3135817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3135979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3136411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3136611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3136631Z 
2023-01-11T22:21:33.3136739Z Running tests...
2023-01-11T22:21:33.3137047Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3137361Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3137632Z   test_all_reduce_sum_cuda_async (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3137852Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34434
2023-01-11T22:21:33.3138054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34435
2023-01-11T22:21:33.3138427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3138601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3138982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3139176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3139548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3139725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3140101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3140291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3140521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3140768Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3141176Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3141577Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3141812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3142043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3142381Z STAGE:2023-01-11 22:00:26 34435:34435 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3142706Z STAGE:2023-01-11 22:00:26 34434:34434 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3143041Z STAGE:2023-01-11 22:00:26 34434:34434 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3143352Z STAGE:2023-01-11 22:00:26 34435:34435 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3143704Z STAGE:2023-01-11 22:00:26 34434:34434 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3144051Z STAGE:2023-01-11 22:00:26 34435:34435 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3144587Z STAGE:2023-01-11 22:00:26 34434:34434 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 22:00:26 34435:34435 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3144607Z 
2023-01-11T22:21:33.3145155Z STAGE:2023-01-11 22:00:26 34435:34435 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:00:26 34434:34434 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3145176Z 
2023-01-11T22:21:33.3145747Z STAGE:2023-01-11 22:00:26 34435:34435 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:00:26 34434:34434 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3145817Z 
2023-01-11T22:21:33.3146153Z STAGE:2023-01-11 22:00:26 34435:34435 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3146484Z STAGE:2023-01-11 22:00:26 34434:34434 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3146866Z STAGE:2023-01-11 22:00:26 34434:34434 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3147193Z STAGE:2023-01-11 22:00:26 34435:34435 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3147537Z STAGE:2023-01-11 22:00:26 34434:34434 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3147861Z STAGE:2023-01-11 22:00:26 34435:34435 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3147963Z ok (6.150s)
2023-01-11T22:21:33.3147983Z 
2023-01-11T22:21:33.3148245Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3148361Z Ran 1 test in 6.151s
2023-01-11T22:21:33.3148381Z 
2023-01-11T22:21:33.3148472Z OK
2023-01-11T22:21:33.3148491Z 
2023-01-11T22:21:33.3148614Z Generating XML reports...
2023-01-11T22:21:33.3149073Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220021.xml
2023-01-11T22:21:33.3149443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3149620Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3149983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3150176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3150195Z 
2023-01-11T22:21:33.3150302Z Running tests...
2023-01-11T22:21:33.3150565Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3150886Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3151161Z   test_all_reduce_sum_cuda_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3151386Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 34550
2023-01-11T22:21:33.3151604Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 34551
2023-01-11T22:21:33.3151959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3152133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3152516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3152707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3153079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3153251Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3153629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3153823Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3154070Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3154298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3154704Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3155101Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3155421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3155657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3156039Z STAGE:2023-01-11 22:00:34 34550:34550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3156368Z STAGE:2023-01-11 22:00:34 34551:34551 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3156706Z STAGE:2023-01-11 22:00:34 34550:34550 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3157041Z STAGE:2023-01-11 22:00:34 34551:34551 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3157369Z STAGE:2023-01-11 22:00:34 34550:34550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3157718Z STAGE:2023-01-11 22:00:34 34551:34551 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3158051Z STAGE:2023-01-11 22:00:34 34550:34550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3158373Z STAGE:2023-01-11 22:00:34 34551:34551 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3158710Z STAGE:2023-01-11 22:00:35 34550:34550 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3159039Z STAGE:2023-01-11 22:00:35 34551:34551 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3159607Z STAGE:2023-01-11 22:00:35 34551:34551 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:00:35 34550:34550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3159628Z 
2023-01-11T22:21:33.3159959Z STAGE:2023-01-11 22:00:35 34551:34551 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3160287Z STAGE:2023-01-11 22:00:35 34550:34550 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3160624Z STAGE:2023-01-11 22:00:35 34550:34550 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3160935Z STAGE:2023-01-11 22:00:35 34551:34551 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3161282Z STAGE:2023-01-11 22:00:35 34550:34550 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3161627Z STAGE:2023-01-11 22:00:35 34551:34551 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3161730Z ok (6.137s)
2023-01-11T22:21:33.3161749Z 
2023-01-11T22:21:33.3162013Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3162124Z Ran 1 test in 6.138s
2023-01-11T22:21:33.3162143Z 
2023-01-11T22:21:33.3162233Z OK
2023-01-11T22:21:33.3162252Z 
2023-01-11T22:21:33.3162376Z Generating XML reports...
2023-01-11T22:21:33.3162815Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220029.xml
2023-01-11T22:21:33.3163195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3163371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3163758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3163952Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3163971Z 
2023-01-11T22:21:33.3164079Z Running tests...
2023-01-11T22:21:33.3164573Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3164908Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3165153Z   test_all_to_all (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T22:21:33.3165174Z 
2023-01-11T22:21:33.3165497Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3165617Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3165636Z 
2023-01-11T22:21:33.3165744Z OK (skipped=1)
2023-01-11T22:21:33.3165763Z 
2023-01-11T22:21:33.3165939Z Generating XML reports...
2023-01-11T22:21:33.3166395Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220038.xml
2023-01-11T22:21:33.3166773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3166951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3167333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3167525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3167545Z 
2023-01-11T22:21:33.3167636Z Running tests...
2023-01-11T22:21:33.3167899Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3168215Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3168472Z   test_all_to_all_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T22:21:33.3168495Z 
2023-01-11T22:21:33.3168754Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3168865Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3168884Z 
2023-01-11T22:21:33.3168990Z OK (skipped=1)
2023-01-11T22:21:33.3169008Z 
2023-01-11T22:21:33.3169131Z Generating XML reports...
2023-01-11T22:21:33.3169563Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220040.xml
2023-01-11T22:21:33.3169937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3170110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3170494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3170688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3170711Z 
2023-01-11T22:21:33.3170818Z Running tests...
2023-01-11T22:21:33.3171077Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3171392Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3171652Z   test_all_to_all_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2023-01-11T22:21:33.3171672Z 
2023-01-11T22:21:33.3171910Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3172023Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3172042Z 
2023-01-11T22:21:33.3172148Z OK (skipped=1)
2023-01-11T22:21:33.3172167Z 
2023-01-11T22:21:33.3172289Z Generating XML reports...
2023-01-11T22:21:33.3172746Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220043.xml
2023-01-11T22:21:33.3173120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3173299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3173683Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3173874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3173894Z 
2023-01-11T22:21:33.3173984Z Running tests...
2023-01-11T22:21:33.3174245Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3174560Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3174885Z   test_all_to_all_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2023-01-11T22:21:33.3174906Z 
2023-01-11T22:21:33.3175171Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3175327Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3175347Z 
2023-01-11T22:21:33.3175453Z OK (skipped=1)
2023-01-11T22:21:33.3175472Z 
2023-01-11T22:21:33.3175593Z Generating XML reports...
2023-01-11T22:21:33.3176029Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220045.xml
2023-01-11T22:21:33.3176400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3176573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3176951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3177147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3177167Z 
2023-01-11T22:21:33.3177278Z Running tests...
2023-01-11T22:21:33.3177541Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3177860Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3178121Z   test_all_to_all_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T22:21:33.3178141Z 
2023-01-11T22:21:33.3178388Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3178500Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3178519Z 
2023-01-11T22:21:33.3178626Z OK (skipped=1)
2023-01-11T22:21:33.3178645Z 
2023-01-11T22:21:33.3178767Z Generating XML reports...
2023-01-11T22:21:33.3179218Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220048.xml
2023-01-11T22:21:33.3179589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3179766Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3180151Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3180342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3180362Z 
2023-01-11T22:21:33.3180451Z Running tests...
2023-01-11T22:21:33.3180714Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3181029Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3181306Z   test_all_to_all_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL supports CUDA all_to_all (0.002s)
2023-01-11T22:21:33.3181326Z 
2023-01-11T22:21:33.3181583Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3181697Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3181717Z 
2023-01-11T22:21:33.3181823Z OK (skipped=1)
2023-01-11T22:21:33.3181842Z 
2023-01-11T22:21:33.3181963Z Generating XML reports...
2023-01-11T22:21:33.3182419Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220050.xml
2023-01-11T22:21:33.3182771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3182945Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3183324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3183516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3183535Z 
2023-01-11T22:21:33.3183641Z Running tests...
2023-01-11T22:21:33.3183902Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3184279Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3184541Z   test_all_to_all_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports all_to_all (0.002s)
2023-01-11T22:21:33.3184603Z 
2023-01-11T22:21:33.3184849Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3184962Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3184981Z 
2023-01-11T22:21:33.3185087Z OK (skipped=1)
2023-01-11T22:21:33.3185105Z 
2023-01-11T22:21:33.3185228Z Generating XML reports...
2023-01-11T22:21:33.3185677Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220052.xml
2023-01-11T22:21:33.3186050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3186225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3186610Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3186805Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3186827Z 
2023-01-11T22:21:33.3186917Z Running tests...
2023-01-11T22:21:33.3187180Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3187496Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3187773Z   test_all_to_all_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T22:21:33.3187792Z 
2023-01-11T22:21:33.3188051Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3188160Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3188179Z 
2023-01-11T22:21:33.3188285Z OK (skipped=1)
2023-01-11T22:21:33.3188304Z 
2023-01-11T22:21:33.3188425Z Generating XML reports...
2023-01-11T22:21:33.3188875Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220055.xml
2023-01-11T22:21:33.3189228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3189409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3189787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3189980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3190000Z 
2023-01-11T22:21:33.3190105Z Running tests...
2023-01-11T22:21:33.3190372Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3190687Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3190977Z   test_all_to_all_single_equal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:21:33.3190997Z 
2023-01-11T22:21:33.3191254Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3191351Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3191370Z 
2023-01-11T22:21:33.3191475Z OK (skipped=1)
2023-01-11T22:21:33.3191494Z 
2023-01-11T22:21:33.3191616Z Generating XML reports...
2023-01-11T22:21:33.3192060Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220057.xml
2023-01-11T22:21:33.3192430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3192607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3192986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3193177Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3193249Z 
2023-01-11T22:21:33.3193344Z Running tests...
2023-01-11T22:21:33.3193610Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3193970Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3194269Z   test_all_to_all_single_equal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:21:33.3194288Z 
2023-01-11T22:21:33.3194547Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3194657Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3194677Z 
2023-01-11T22:21:33.3194781Z OK (skipped=1)
2023-01-11T22:21:33.3194800Z 
2023-01-11T22:21:33.3194921Z Generating XML reports...
2023-01-11T22:21:33.3195373Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220100.xml
2023-01-11T22:21:33.3195729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3195907Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3196291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3196486Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3196506Z 
2023-01-11T22:21:33.3196613Z Running tests...
2023-01-11T22:21:33.3196875Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3197187Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3197482Z   test_all_to_all_single_equal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T22:21:33.3197503Z 
2023-01-11T22:21:33.3197762Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3197857Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3197877Z 
2023-01-11T22:21:33.3197985Z OK (skipped=1)
2023-01-11T22:21:33.3198004Z 
2023-01-11T22:21:33.3198125Z Generating XML reports...
2023-01-11T22:21:33.3198582Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220102.xml
2023-01-11T22:21:33.3198952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3199128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3199506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3199697Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3199716Z 
2023-01-11T22:21:33.3199822Z Running tests...
2023-01-11T22:21:33.3200063Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3200377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3200686Z   test_all_to_all_single_equal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T22:21:33.3200708Z 
2023-01-11T22:21:33.3200969Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3201081Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3201100Z 
2023-01-11T22:21:33.3201205Z OK (skipped=1)
2023-01-11T22:21:33.3201224Z 
2023-01-11T22:21:33.3201346Z Generating XML reports...
2023-01-11T22:21:33.3201791Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220104.xml
2023-01-11T22:21:33.3202161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3202320Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3202748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3202978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3202998Z 
2023-01-11T22:21:33.3203102Z Running tests...
2023-01-11T22:21:33.3203366Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3203682Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3203981Z   test_all_to_all_single_equal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:21:33.3204001Z 
2023-01-11T22:21:33.3204474Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3204572Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3204610Z 
2023-01-11T22:21:33.3204699Z OK (skipped=1)
2023-01-11T22:21:33.3204718Z 
2023-01-11T22:21:33.3204846Z Generating XML reports...
2023-01-11T22:21:33.3205307Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220107.xml
2023-01-11T22:21:33.3205685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3205861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3206244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3206436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3206456Z 
2023-01-11T22:21:33.3206563Z Running tests...
2023-01-11T22:21:33.3206808Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3207121Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3207431Z   test_all_to_all_single_equal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T22:21:33.3207451Z 
2023-01-11T22:21:33.3207715Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3207826Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3207846Z 
2023-01-11T22:21:33.3207952Z OK (skipped=1)
2023-01-11T22:21:33.3207971Z 
2023-01-11T22:21:33.3208094Z Generating XML reports...
2023-01-11T22:21:33.3208543Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220109.xml
2023-01-11T22:21:33.3208912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3209070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3209450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3209646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3209665Z 
2023-01-11T22:21:33.3209773Z Running tests...
2023-01-11T22:21:33.3210036Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3210389Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3210684Z   test_all_to_all_single_equal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:21:33.3210703Z 
2023-01-11T22:21:33.3210964Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3211073Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3211092Z 
2023-01-11T22:21:33.3211181Z OK (skipped=1)
2023-01-11T22:21:33.3211199Z 
2023-01-11T22:21:33.3211322Z Generating XML reports...
2023-01-11T22:21:33.3211847Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220111.xml
2023-01-11T22:21:33.3212233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3212470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3212851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3213043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3213062Z 
2023-01-11T22:21:33.3213169Z Running tests...
2023-01-11T22:21:33.3213413Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3213731Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3214034Z   test_all_to_all_single_equal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T22:21:33.3214053Z 
2023-01-11T22:21:33.3214315Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3214426Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3214445Z 
2023-01-11T22:21:33.3214553Z OK (skipped=1)
2023-01-11T22:21:33.3214572Z 
2023-01-11T22:21:33.3214694Z Generating XML reports...
2023-01-11T22:21:33.3215144Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220114.xml
2023-01-11T22:21:33.3215515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3215673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3216058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3216250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3216270Z 
2023-01-11T22:21:33.3216378Z Running tests...
2023-01-11T22:21:33.3216645Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3216956Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3217247Z   test_all_to_all_single_unequal_split (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:21:33.3217266Z 
2023-01-11T22:21:33.3217523Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3217633Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3217652Z 
2023-01-11T22:21:33.3217741Z OK (skipped=1)
2023-01-11T22:21:33.3217760Z 
2023-01-11T22:21:33.3217884Z Generating XML reports...
2023-01-11T22:21:33.3218331Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220116.xml
2023-01-11T22:21:33.3218702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3218882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3219260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3219453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3219473Z 
2023-01-11T22:21:33.3219582Z Running tests...
2023-01-11T22:21:33.3219838Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3220132Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3220433Z   test_all_to_all_single_unequal_split_complex (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:21:33.3220453Z 
2023-01-11T22:21:33.3220712Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3220822Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3220841Z 
2023-01-11T22:21:33.3221000Z OK (skipped=1)
2023-01-11T22:21:33.3221021Z 
2023-01-11T22:21:33.3221146Z Generating XML reports...
2023-01-11T22:21:33.3221602Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220119.xml
2023-01-11T22:21:33.3222020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3222197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3222561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3222754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3222774Z 
2023-01-11T22:21:33.3222880Z Running tests...
2023-01-11T22:21:33.3223141Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3223459Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3223755Z   test_all_to_all_single_unequal_split_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T22:21:33.3223778Z 
2023-01-11T22:21:33.3224041Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3224151Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3224170Z 
2023-01-11T22:21:33.3224258Z OK (skipped=1)
2023-01-11T22:21:33.3224295Z 
2023-01-11T22:21:33.3224400Z Generating XML reports...
2023-01-11T22:21:33.3224847Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220121.xml
2023-01-11T22:21:33.3225221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3225397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3225779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3225972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3225994Z 
2023-01-11T22:21:33.3226102Z Running tests...
2023-01-11T22:21:33.3226365Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3226660Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3226969Z   test_all_to_all_single_unequal_split_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T22:21:33.3226989Z 
2023-01-11T22:21:33.3227245Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3227355Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3227375Z 
2023-01-11T22:21:33.3227480Z OK (skipped=1)
2023-01-11T22:21:33.3227498Z 
2023-01-11T22:21:33.3227620Z Generating XML reports...
2023-01-11T22:21:33.3228070Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220123.xml
2023-01-11T22:21:33.3228444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3228624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3228985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3229175Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3229194Z 
2023-01-11T22:21:33.3229302Z Running tests...
2023-01-11T22:21:33.3229563Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3229880Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3230233Z   test_all_to_all_single_unequal_split_full_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:21:33.3230255Z 
2023-01-11T22:21:33.3230518Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3230673Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3230692Z 
2023-01-11T22:21:33.3230797Z OK (skipped=1)
2023-01-11T22:21:33.3230816Z 
2023-01-11T22:21:33.3230920Z Generating XML reports...
2023-01-11T22:21:33.3231372Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220126.xml
2023-01-11T22:21:33.3231741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3231919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3232305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3232502Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3232521Z 
2023-01-11T22:21:33.3232629Z Running tests...
2023-01-11T22:21:33.3232894Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3233196Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3233511Z   test_all_to_all_single_unequal_split_full_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T22:21:33.3233530Z 
2023-01-11T22:21:33.3233788Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3233898Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3233918Z 
2023-01-11T22:21:33.3234024Z OK (skipped=1)
2023-01-11T22:21:33.3234043Z 
2023-01-11T22:21:33.3234165Z Generating XML reports...
2023-01-11T22:21:33.3234617Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220128.xml
2023-01-11T22:21:33.3234992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3235168Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3235548Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3235724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3235744Z 
2023-01-11T22:21:33.3235852Z Running tests...
2023-01-11T22:21:33.3236109Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3236423Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3236719Z   test_all_to_all_single_unequal_split_group (__main__.TestDistBackendWithSpawn) ... skip: Only MPI supports CPU all_to_all_single (0.002s)
2023-01-11T22:21:33.3236739Z 
2023-01-11T22:21:33.3237002Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3237113Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3237133Z 
2023-01-11T22:21:33.3237240Z OK (skipped=1)
2023-01-11T22:21:33.3237260Z 
2023-01-11T22:21:33.3237369Z Generating XML reports...
2023-01-11T22:21:33.3237821Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220131.xml
2023-01-11T22:21:33.3238193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3238369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3238751Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3238943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3238963Z 
2023-01-11T22:21:33.3239072Z Running tests...
2023-01-11T22:21:33.3239388Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3239713Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3240003Z   test_all_to_all_single_unequal_split_group_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA all_to_all_single (0.002s)
2023-01-11T22:21:33.3240086Z 
2023-01-11T22:21:33.3240335Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3240446Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3240464Z 
2023-01-11T22:21:33.3240570Z OK (skipped=1)
2023-01-11T22:21:33.3240589Z 
2023-01-11T22:21:33.3240712Z Generating XML reports...
2023-01-11T22:21:33.3241160Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220133.xml
2023-01-11T22:21:33.3241528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3241707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3242090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3242267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3242287Z 
2023-01-11T22:21:33.3242396Z Running tests...
2023-01-11T22:21:33.3242656Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3242969Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3243236Z   test_average_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3243458Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35458
2023-01-11T22:21:33.3243679Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35459
2023-01-11T22:21:33.3244053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3244445Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3244851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3245052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3245416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3245589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3245962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3246154Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3246403Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3246652Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3247038Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3247444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3247678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3247909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3248147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.3248389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.3248877Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.3249291Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.3249452Z ok (5.645s)
2023-01-11T22:21:33.3249472Z 
2023-01-11T22:21:33.3249719Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3249830Z Ran 1 test in 5.645s
2023-01-11T22:21:33.3249849Z 
2023-01-11T22:21:33.3249942Z OK
2023-01-11T22:21:33.3249961Z 
2023-01-11T22:21:33.3250088Z Generating XML reports...
2023-01-11T22:21:33.3250538Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220135.xml
2023-01-11T22:21:33.3250911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3251088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3251473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3251650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3251691Z 
2023-01-11T22:21:33.3251783Z Running tests...
2023-01-11T22:21:33.3252043Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3252358Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3252619Z   test_backend_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3252838Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35579
2023-01-11T22:21:33.3253058Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35580
2023-01-11T22:21:33.3253431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3253611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3253974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3254170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3254535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3254709Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3255090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3255281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3255528Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3255779Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3256182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3256566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3256798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3257028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3257177Z skip: Need at least 3 CUDA devices (4.157s)
2023-01-11T22:21:33.3257197Z 
2023-01-11T22:21:33.3257460Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3257569Z Ran 1 test in 4.157s
2023-01-11T22:21:33.3257588Z 
2023-01-11T22:21:33.3257694Z OK (skipped=1)
2023-01-11T22:21:33.3257713Z 
2023-01-11T22:21:33.3257837Z Generating XML reports...
2023-01-11T22:21:33.3258322Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220144.xml
2023-01-11T22:21:33.3258703Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3258924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3259304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3259496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3259515Z 
2023-01-11T22:21:33.3259623Z Running tests...
2023-01-11T22:21:33.3259883Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3260197Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3260449Z   test_backend_group (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 3 (0.002s)
2023-01-11T22:21:33.3260473Z 
2023-01-11T22:21:33.3260727Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3260838Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3260860Z 
2023-01-11T22:21:33.3260967Z OK (skipped=1)
2023-01-11T22:21:33.3260985Z 
2023-01-11T22:21:33.3261107Z Generating XML reports...
2023-01-11T22:21:33.3261559Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220150.xml
2023-01-11T22:21:33.3261931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3262105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3262486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3262677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3262697Z 
2023-01-11T22:21:33.3262790Z Running tests...
2023-01-11T22:21:33.3263051Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3263364Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3263612Z   test_barrier (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3263834Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35721
2023-01-11T22:21:33.3264050Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35722
2023-01-11T22:21:33.3264422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3264595Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3264955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3265149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3265517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3265696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3266071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3266260Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3266507Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3266756Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3267161Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3267595Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3267833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3268104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3268205Z ok (5.055s)
2023-01-11T22:21:33.3268224Z 
2023-01-11T22:21:33.3268490Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3268599Z Ran 1 test in 5.055s
2023-01-11T22:21:33.3268618Z 
2023-01-11T22:21:33.3268710Z OK
2023-01-11T22:21:33.3268729Z 
2023-01-11T22:21:33.3268852Z Generating XML reports...
2023-01-11T22:21:33.3269287Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220153.xml
2023-01-11T22:21:33.3269660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3269838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3270222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3270416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3270435Z 
2023-01-11T22:21:33.3270543Z Running tests...
2023-01-11T22:21:33.3270805Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3271120Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3271373Z   test_barrier_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3271575Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35830
2023-01-11T22:21:33.3271790Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35831
2023-01-11T22:21:33.3272161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3272339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3272723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3272914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3273277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3273448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3273802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3273993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3274247Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3274493Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3274895Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3275293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3275525Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3275752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3275853Z ok (5.878s)
2023-01-11T22:21:33.3275873Z 
2023-01-11T22:21:33.3276119Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3276231Z Ran 1 test in 5.879s
2023-01-11T22:21:33.3276251Z 
2023-01-11T22:21:33.3276342Z OK
2023-01-11T22:21:33.3276361Z 
2023-01-11T22:21:33.3276534Z Generating XML reports...
2023-01-11T22:21:33.3276996Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220200.xml
2023-01-11T22:21:33.3277423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3277598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3277980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3278171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3278191Z 
2023-01-11T22:21:33.3278281Z Running tests...
2023-01-11T22:21:33.3278540Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3278854Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3279121Z   test_barrier_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3279343Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 35941
2023-01-11T22:21:33.3279564Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 35942
2023-01-11T22:21:33.3279932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3280107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3280470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3280662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3281026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3281206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3281582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3281775Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3282023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3282273Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3282676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3283057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3283289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3283521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3283762Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.3284005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.3284628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.3285038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.3285141Z ok (5.005s)
2023-01-11T22:21:33.3285161Z 
2023-01-11T22:21:33.3285425Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3285518Z Ran 1 test in 5.005s
2023-01-11T22:21:33.3285537Z 
2023-01-11T22:21:33.3285630Z OK
2023-01-11T22:21:33.3285649Z 
2023-01-11T22:21:33.3285772Z Generating XML reports...
2023-01-11T22:21:33.3286303Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220209.xml
2023-01-11T22:21:33.3286690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3286927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3287311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3287505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3287525Z 
2023-01-11T22:21:33.3287615Z Running tests...
2023-01-11T22:21:33.3287875Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3288190Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3288461Z   test_barrier_full_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3288686Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36056
2023-01-11T22:21:33.3288908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36057
2023-01-11T22:21:33.3289282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3289459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3289839Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3290013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3290376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3290547Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3290932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3291125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3291375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3291620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3292022Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3292460Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3292695Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3292924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3293086Z skip: Skipped due to small world size. (4.133s)
2023-01-11T22:21:33.3293107Z 
2023-01-11T22:21:33.3293372Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3293487Z Ran 1 test in 4.134s
2023-01-11T22:21:33.3293507Z 
2023-01-11T22:21:33.3293613Z OK (skipped=1)
2023-01-11T22:21:33.3293632Z 
2023-01-11T22:21:33.3293755Z Generating XML reports...
2023-01-11T22:21:33.3294208Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220216.xml
2023-01-11T22:21:33.3294562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3294739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3295120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3295311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3295381Z 
2023-01-11T22:21:33.3295495Z Running tests...
2023-01-11T22:21:33.3295761Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3296163Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3296416Z   test_barrier_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3296618Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36165
2023-01-11T22:21:33.3296838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36166
2023-01-11T22:21:33.3297208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3297385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3297766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3297961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3298327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3298505Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3298880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3299051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3299298Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3299544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3299947Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3300344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3300576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3300809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3300968Z skip: Skipped due to small world size. (4.239s)
2023-01-11T22:21:33.3300987Z 
2023-01-11T22:21:33.3301249Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3301342Z Ran 1 test in 4.239s
2023-01-11T22:21:33.3301361Z 
2023-01-11T22:21:33.3301468Z OK (skipped=1)
2023-01-11T22:21:33.3301487Z 
2023-01-11T22:21:33.3301609Z Generating XML reports...
2023-01-11T22:21:33.3302064Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220223.xml
2023-01-11T22:21:33.3302441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3302616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3302996Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3303191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3303210Z 
2023-01-11T22:21:33.3303318Z Running tests...
2023-01-11T22:21:33.3303563Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3303877Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3304140Z   test_barrier_group_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3304359Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36274
2023-01-11T22:21:33.3304628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36275
2023-01-11T22:21:33.3305015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3305237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3305621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3305795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3306159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3306333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3306707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3306894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3307146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3307393Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3307801Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3308198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3308412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3308641Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3308798Z skip: Skipped due to small world size. (4.111s)
2023-01-11T22:21:33.3308818Z 
2023-01-11T22:21:33.3309084Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3309198Z Ran 1 test in 4.112s
2023-01-11T22:21:33.3309218Z 
2023-01-11T22:21:33.3309326Z OK (skipped=1)
2023-01-11T22:21:33.3309345Z 
2023-01-11T22:21:33.3309468Z Generating XML reports...
2023-01-11T22:21:33.3309927Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220230.xml
2023-01-11T22:21:33.3310325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3310508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3310889Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3311080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3311099Z 
2023-01-11T22:21:33.3311209Z Running tests...
2023-01-11T22:21:33.3311474Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3311793Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3312066Z   test_barrier_timeout_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3312290Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36383
2023-01-11T22:21:33.3312490Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36384
2023-01-11T22:21:33.3312860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3313035Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3313414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3313605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3314033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3314214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3314597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3314815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3315064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3315308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3315711Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3316108Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3316346Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3316577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3316820Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.3317062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.3317444Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.3317840Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.3317942Z ok (5.317s)
2023-01-11T22:21:33.3317962Z 
2023-01-11T22:21:33.3318225Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3318339Z Ran 1 test in 5.318s
2023-01-11T22:21:33.3318358Z 
2023-01-11T22:21:33.3318451Z OK
2023-01-11T22:21:33.3318474Z 
2023-01-11T22:21:33.3318597Z Generating XML reports...
2023-01-11T22:21:33.3319048Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220236.xml
2023-01-11T22:21:33.3319424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3319583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3319964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3320156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3320175Z 
2023-01-11T22:21:33.3320283Z Running tests...
2023-01-11T22:21:33.3320546Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3320864Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3321137Z   test_barrier_timeout_global (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3321358Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36498
2023-01-11T22:21:33.3321561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36499
2023-01-11T22:21:33.3321933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3322109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3322489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3322681Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3323047Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3323272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3323662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3323900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3324129Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3324598Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3325016Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3325414Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3325647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3325882Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3326126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3326375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3326775Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3327152Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3327254Z ok (5.313s)
2023-01-11T22:21:33.3327274Z 
2023-01-11T22:21:33.3327538Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3327650Z Ran 1 test in 5.314s
2023-01-11T22:21:33.3327670Z 
2023-01-11T22:21:33.3327763Z OK
2023-01-11T22:21:33.3327782Z 
2023-01-11T22:21:33.3327906Z Generating XML reports...
2023-01-11T22:21:33.3328365Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220244.xml
2023-01-11T22:21:33.3328742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3328919Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3329285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3329476Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3329496Z 
2023-01-11T22:21:33.3329604Z Running tests...
2023-01-11T22:21:33.3329866Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3330182Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3330452Z   test_barrier_timeout_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3330672Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36613
2023-01-11T22:21:33.3330893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36614
2023-01-11T22:21:33.3331245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3331421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3331803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3331994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3332361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3332534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3332989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3333191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3333477Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3333721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3334125Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3334520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3334752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3334981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3335143Z skip: Skipped due to small world size. (4.216s)
2023-01-11T22:21:33.3335163Z 
2023-01-11T22:21:33.3335431Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3335545Z Ran 1 test in 4.216s
2023-01-11T22:21:33.3335565Z 
2023-01-11T22:21:33.3335654Z OK (skipped=1)
2023-01-11T22:21:33.3335691Z 
2023-01-11T22:21:33.3335798Z Generating XML reports...
2023-01-11T22:21:33.3336253Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220252.xml
2023-01-11T22:21:33.3336625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3336800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3337182Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3337375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3337395Z 
2023-01-11T22:21:33.3337503Z Running tests...
2023-01-11T22:21:33.3337766Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3338065Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3338335Z   test_batch_isend_irecv_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3338556Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36722
2023-01-11T22:21:33.3338774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36723
2023-01-11T22:21:33.3339147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3339323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3339711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3339903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3340249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3340428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3340801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3340990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3341237Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3341483Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3341933Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3342343Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3342624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3342836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3342940Z ok (4.234s)
2023-01-11T22:21:33.3342960Z 
2023-01-11T22:21:33.3343224Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3343336Z Ran 1 test in 4.234s
2023-01-11T22:21:33.3343356Z 
2023-01-11T22:21:33.3343447Z OK
2023-01-11T22:21:33.3343467Z 
2023-01-11T22:21:33.3343590Z Generating XML reports...
2023-01-11T22:21:33.3344042Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220259.xml
2023-01-11T22:21:33.3344417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3344593Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3344958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3345150Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3345170Z 
2023-01-11T22:21:33.3345278Z Running tests...
2023-01-11T22:21:33.3345543Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3345862Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3346135Z   test_batch_isend_irecv_gloo_tags (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3346356Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 36831
2023-01-11T22:21:33.3346577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 36832
2023-01-11T22:21:33.3346934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3347115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3347498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3347689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3348052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3348225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3348599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3348788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3349037Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3349264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3349669Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3350067Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3350299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3350528Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3350630Z ok (4.263s)
2023-01-11T22:21:33.3350650Z 
2023-01-11T22:21:33.3350916Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3351027Z Ran 1 test in 4.263s
2023-01-11T22:21:33.3351095Z 
2023-01-11T22:21:33.3351173Z OK
2023-01-11T22:21:33.3351212Z 
2023-01-11T22:21:33.3351317Z Generating XML reports...
2023-01-11T22:21:33.3351772Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220305.xml
2023-01-11T22:21:33.3352190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3352369Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3352748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3352940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3352960Z 
2023-01-11T22:21:33.3353067Z Running tests...
2023-01-11T22:21:33.3353329Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3353629Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3353908Z   test_batch_isend_irecv_mixed_backend_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T22:21:33.3353931Z 
2023-01-11T22:21:33.3354192Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3354302Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3354322Z 
2023-01-11T22:21:33.3354428Z OK (skipped=1)
2023-01-11T22:21:33.3354448Z 
2023-01-11T22:21:33.3354569Z Generating XML reports...
2023-01-11T22:21:33.3355020Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220312.xml
2023-01-11T22:21:33.3355390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3355568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3355929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3356123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3356146Z 
2023-01-11T22:21:33.3356254Z Running tests...
2023-01-11T22:21:33.3356518Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3356834Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3357096Z   test_batch_isend_irecv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.003s)
2023-01-11T22:21:33.3357115Z 
2023-01-11T22:21:33.3357372Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3357483Z Ran 1 test in 0.003s
2023-01-11T22:21:33.3357502Z 
2023-01-11T22:21:33.3357591Z OK (skipped=1)
2023-01-11T22:21:33.3357629Z 
2023-01-11T22:21:33.3357734Z Generating XML reports...
2023-01-11T22:21:33.3358191Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220315.xml
2023-01-11T22:21:33.3358565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3358748Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3359133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3359326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3359345Z 
2023-01-11T22:21:33.3359452Z Running tests...
2023-01-11T22:21:33.3359715Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3360011Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3360286Z   test_batch_isend_irecv_no_rank_zero_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T22:21:33.3360356Z 
2023-01-11T22:21:33.3360625Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3360736Z Ran 1 test in 0.003s
2023-01-11T22:21:33.3360796Z 
2023-01-11T22:21:33.3360905Z OK (skipped=1)
2023-01-11T22:21:33.3360923Z 
2023-01-11T22:21:33.3361049Z Generating XML reports...
2023-01-11T22:21:33.3361503Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220317.xml
2023-01-11T22:21:33.3361875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3362052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3362417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3362611Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3362630Z 
2023-01-11T22:21:33.3362741Z Running tests...
2023-01-11T22:21:33.3363002Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3363318Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3363582Z   test_batch_isend_irecv_op_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T22:21:33.3363602Z 
2023-01-11T22:21:33.3363858Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3363968Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3363987Z 
2023-01-11T22:21:33.3364093Z OK (skipped=1)
2023-01-11T22:21:33.3364112Z 
2023-01-11T22:21:33.3364524Z Generating XML reports...
2023-01-11T22:21:33.3365001Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220319.xml
2023-01-11T22:21:33.3365379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3365556Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3365938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3366137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3366157Z 
2023-01-11T22:21:33.3366266Z Running tests...
2023-01-11T22:21:33.3366529Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3366827Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3367098Z   test_batch_isend_irecv_op_list_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T22:21:33.3367118Z 
2023-01-11T22:21:33.3367378Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3367489Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3367509Z 
2023-01-11T22:21:33.3367618Z OK (skipped=1)
2023-01-11T22:21:33.3367637Z 
2023-01-11T22:21:33.3367762Z Generating XML reports...
2023-01-11T22:21:33.3368210Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220322.xml
2023-01-11T22:21:33.3368584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3368762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3369122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3369314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3369333Z 
2023-01-11T22:21:33.3369443Z Running tests...
2023-01-11T22:21:33.3369704Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3370104Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3370396Z   test_batch_isend_irecv_ring_exchange_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T22:21:33.3370467Z 
2023-01-11T22:21:33.3370734Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3370848Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3370867Z 
2023-01-11T22:21:33.3370975Z OK (skipped=1)
2023-01-11T22:21:33.3370993Z 
2023-01-11T22:21:33.3371098Z Generating XML reports...
2023-01-11T22:21:33.3371552Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220324.xml
2023-01-11T22:21:33.3371922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3372100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3372484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3372679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3372698Z 
2023-01-11T22:21:33.3372809Z Running tests...
2023-01-11T22:21:33.3373074Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3373367Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3373632Z   test_batch_isend_irecv_self_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T22:21:33.3373651Z 
2023-01-11T22:21:33.3373911Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3374023Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3374042Z 
2023-01-11T22:21:33.3374150Z OK (skipped=1)
2023-01-11T22:21:33.3374169Z 
2023-01-11T22:21:33.3374294Z Generating XML reports...
2023-01-11T22:21:33.3374748Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220327.xml
2023-01-11T22:21:33.3375120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3375298Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3375660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3375852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3375871Z 
2023-01-11T22:21:33.3375981Z Running tests...
2023-01-11T22:21:33.3376241Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3376555Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3376822Z   test_batch_isend_irecv_tensor_err (__main__.TestDistBackendWithSpawn) ... skip: NCCL Batch Send Recv Only (0.002s)
2023-01-11T22:21:33.3376842Z 
2023-01-11T22:21:33.3377105Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3377217Z Ran 1 test in 0.002s
2023-01-11T22:21:33.3377235Z 
2023-01-11T22:21:33.3377346Z OK (skipped=1)
2023-01-11T22:21:33.3377365Z 
2023-01-11T22:21:33.3377470Z Generating XML reports...
2023-01-11T22:21:33.3377914Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220329.xml
2023-01-11T22:21:33.3378284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3378459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3378836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3379028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3379048Z 
2023-01-11T22:21:33.3379157Z Running tests...
2023-01-11T22:21:33.3379484Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3379810Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3380088Z   test_broadcast (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3380310Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37204
2023-01-11T22:21:33.3380529Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37205
2023-01-11T22:21:33.3380901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3381077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3381457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3381652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3382016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3382174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3382549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3382740Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3382988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3383236Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3383637Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3384043Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3384276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3384512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3384833Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3385161Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3385501Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3385841Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3386195Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3386545Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3386874Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3387199Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3387534Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3387862Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3388194Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3388540Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3388868Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3389247Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3389592Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3389990Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3390323Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3390666Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3390976Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3391298Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3391631Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3391978Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3392311Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3392657Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3392982Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3393301Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3393630Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3393954Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3394286Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3394631Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3394958Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3395283Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3395614Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3395955Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3396282Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3396624Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3396936Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3397257Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3397594Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3397938Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3398270Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3398614Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3398939Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3399260Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3399648Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3399981Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3400362Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3400705Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3401031Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3401353Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3401686Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3402026Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3402360Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3402700Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3403012Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3403331Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3403662Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3404006Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3404512Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3404868Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3405198Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3405523Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3405859Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3406182Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3406510Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3406852Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3407176Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3407502Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3407836Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3408177Z STAGE:2023-01-11 22:03:35 37205:37205 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3408512Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3408836Z STAGE:2023-01-11 22:03:35 37204:37204 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3408940Z ok (4.218s)
2023-01-11T22:21:33.3408959Z 
2023-01-11T22:21:33.3409222Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3409333Z Ran 1 test in 4.218s
2023-01-11T22:21:33.3409352Z 
2023-01-11T22:21:33.3409445Z OK
2023-01-11T22:21:33.3409464Z 
2023-01-11T22:21:33.3409588Z Generating XML reports...
2023-01-11T22:21:33.3410159Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220331.xml
2023-01-11T22:21:33.3410554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3410793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3411161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3411355Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3411375Z 
2023-01-11T22:21:33.3411483Z Running tests...
2023-01-11T22:21:33.3411747Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3412059Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3412316Z   test_broadcast_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3413077Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81028 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.631s)
2023-01-11T22:21:33.3413101Z 
2023-01-11T22:21:33.3413360Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3413471Z Ran 1 test in 1.631s
2023-01-11T22:21:33.3413490Z 
2023-01-11T22:21:33.3413579Z OK (skipped=1)
2023-01-11T22:21:33.3413620Z 
2023-01-11T22:21:33.3413725Z Generating XML reports...
2023-01-11T22:21:33.3414180Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220338.xml
2023-01-11T22:21:33.3414555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3414734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3415119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3415311Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3415333Z 
2023-01-11T22:21:33.3415444Z Running tests...
2023-01-11T22:21:33.3415708Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3416007Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3416276Z   test_broadcast_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3416500Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37351
2023-01-11T22:21:33.3416719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37352
2023-01-11T22:21:33.3417098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3417275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3417660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3417857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3418228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3418383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3418765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3418960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3419209Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3419511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3419927Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3420379Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3420612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3420823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3421064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.3421308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.3421709Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.3422110Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.3422449Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3422779Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3423327Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3423348Z 
2023-01-11T22:21:33.3423922Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3423943Z 
2023-01-11T22:21:33.3424272Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3424592Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3424911Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3425471Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3425490Z 
2023-01-11T22:21:33.3425840Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3426166Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3426487Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3426820Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3427157Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3427506Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3427851Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3428175Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3428474Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3428806Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3429135Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3429534Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3429892Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3430270Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3430599Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3430933Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3431262Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3431588Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3431933Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3432260Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3432578Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3432916Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3433471Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3433491Z 
2023-01-11T22:21:33.3433836Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3434156Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3434478Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3434805Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3435115Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3435461Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3435803Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3436129Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3436452Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3436788Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3437135Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3437465Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3437814Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3438120Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3438440Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3438770Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3439098Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3439443Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3439837Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3440177Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3440553Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3440880Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3441416Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3441455Z 
2023-01-11T22:21:33.3441779Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3442103Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3442425Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3442755Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3443083Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3443424Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3443769Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3444093Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3444577Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3444922Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3445249Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3445595Z STAGE:2023-01-11 22:03:46 37352:37352 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3445941Z STAGE:2023-01-11 22:03:46 37351:37351 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3446044Z ok (4.199s)
2023-01-11T22:21:33.3446064Z 
2023-01-11T22:21:33.3446329Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3446441Z Ran 1 test in 4.200s
2023-01-11T22:21:33.3446461Z 
2023-01-11T22:21:33.3446553Z OK
2023-01-11T22:21:33.3446572Z 
2023-01-11T22:21:33.3446679Z Generating XML reports...
2023-01-11T22:21:33.3447140Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220342.xml
2023-01-11T22:21:33.3447521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3447700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3448090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3448285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3448305Z 
2023-01-11T22:21:33.3448416Z Running tests...
2023-01-11T22:21:33.3448677Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3448978Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3449239Z   test_broadcast_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3449461Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37470
2023-01-11T22:21:33.3449798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37471
2023-01-11T22:21:33.3450187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3450421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3450808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3451000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3451369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3451526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3451903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3452091Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3452344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3452593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3453001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3453401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3453638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3453869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3454011Z skip: Skipped due to small world size. (4.131s)
2023-01-11T22:21:33.3454031Z 
2023-01-11T22:21:33.3456818Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3456941Z Ran 1 test in 4.131s
2023-01-11T22:21:33.3456961Z 
2023-01-11T22:21:33.3457070Z OK (skipped=1)
2023-01-11T22:21:33.3457089Z 
2023-01-11T22:21:33.3457214Z Generating XML reports...
2023-01-11T22:21:33.3457676Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220349.xml
2023-01-11T22:21:33.3458051Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3458226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3458606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3458779Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3458798Z 
2023-01-11T22:21:33.3458906Z Running tests...
2023-01-11T22:21:33.3459167Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3459486Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3459751Z   test_broadcast_multigpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3459974Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37579
2023-01-11T22:21:33.3460193Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37580
2023-01-11T22:21:33.3460565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3460723Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3461102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3461293Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3461714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3461897Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3462278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3462516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3462765Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3463011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3463398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3463796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3464032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3464265Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3465050Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1478: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T22:21:33.3465163Z   warnings.warn(
2023-01-11T22:21:33.3465936Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:1478: UserWarning: torch.distributed.broadcast_multigpu will be deprecated. If you must use it, please revisit our documentation later at https://pytorch.org/docs/master/distributed.html#multi-gpu-collective-functions
2023-01-11T22:21:33.3466045Z   warnings.warn(
2023-01-11T22:21:33.3466145Z ok (5.134s)
2023-01-11T22:21:33.3466165Z 
2023-01-11T22:21:33.3466430Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3466524Z Ran 1 test in 5.134s
2023-01-11T22:21:33.3466543Z 
2023-01-11T22:21:33.3466635Z OK
2023-01-11T22:21:33.3466657Z 
2023-01-11T22:21:33.3466781Z Generating XML reports...
2023-01-11T22:21:33.3467236Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220356.xml
2023-01-11T22:21:33.3467608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3467786Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3468167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3468359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3468378Z 
2023-01-11T22:21:33.3468469Z Running tests...
2023-01-11T22:21:33.3468740Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3469058Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3469329Z   test_broadcast_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3470082Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82847 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.587s)
2023-01-11T22:21:33.3470102Z 
2023-01-11T22:21:33.3470361Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3470473Z Ran 1 test in 1.587s
2023-01-11T22:21:33.3470492Z 
2023-01-11T22:21:33.3470599Z OK (skipped=1)
2023-01-11T22:21:33.3470617Z 
2023-01-11T22:21:33.3470742Z Generating XML reports...
2023-01-11T22:21:33.3471245Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220403.xml
2023-01-11T22:21:33.3471613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3471840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3472225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3472417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3472437Z 
2023-01-11T22:21:33.3472545Z Running tests...
2023-01-11T22:21:33.3472807Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3473123Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3473445Z   test_compute_bucket_assignment_by_size_sparse_error_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3474194Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85012 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.633s)
2023-01-11T22:21:33.3474218Z 
2023-01-11T22:21:33.3474477Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3474572Z Ran 1 test in 1.634s
2023-01-11T22:21:33.3474591Z 
2023-01-11T22:21:33.3474699Z OK (skipped=1)
2023-01-11T22:21:33.3474718Z 
2023-01-11T22:21:33.3474842Z Generating XML reports...
2023-01-11T22:21:33.3475292Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220408.xml
2023-01-11T22:21:33.3475670Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3475846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3476227Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3476426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3476445Z 
2023-01-11T22:21:33.3476535Z Running tests...
2023-01-11T22:21:33.3476800Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3477118Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3477442Z   test_compute_bucket_assignment_by_size_sparse_error_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3478191Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/85339 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.640s)
2023-01-11T22:21:33.3478212Z 
2023-01-11T22:21:33.3478473Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3478583Z Ran 1 test in 1.640s
2023-01-11T22:21:33.3478603Z 
2023-01-11T22:21:33.3478709Z OK (skipped=1)
2023-01-11T22:21:33.3478728Z 
2023-01-11T22:21:33.3478850Z Generating XML reports...
2023-01-11T22:21:33.3479299Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220412.xml
2023-01-11T22:21:33.3479652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3479830Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3480212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3480456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3480477Z 
2023-01-11T22:21:33.3480588Z Running tests...
2023-01-11T22:21:33.3480896Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3481214Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3481492Z   test_ddp_apply_optim_in_backward (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3481695Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37792
2023-01-11T22:21:33.3481912Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37793
2023-01-11T22:21:33.3482284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3482461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3482847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3483041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3483410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3483582Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3483956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3484128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3484559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3484814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3485229Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3485624Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3485861Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3486092Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3486878Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T22:21:33.3486992Z   warnings.warn(
2023-01-11T22:21:33.3487776Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T22:21:33.3487873Z   warnings.warn(
2023-01-11T22:21:33.3488111Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3488342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3488578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3488812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3489045Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3489272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3489570Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3489812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3489952Z ok (6.943s)
2023-01-11T22:21:33.3489971Z 
2023-01-11T22:21:33.3490241Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3490353Z Ran 1 test in 6.943s
2023-01-11T22:21:33.3490372Z 
2023-01-11T22:21:33.3490465Z OK
2023-01-11T22:21:33.3490484Z 
2023-01-11T22:21:33.3490609Z Generating XML reports...
2023-01-11T22:21:33.3491065Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220416.xml
2023-01-11T22:21:33.3491435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3491611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3491976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3492170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3492192Z 
2023-01-11T22:21:33.3492300Z Running tests...
2023-01-11T22:21:33.3492559Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3492876Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3493186Z   test_ddp_apply_optim_in_backward_grad_as_bucket_view_false (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3493409Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 37907
2023-01-11T22:21:33.3493627Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 37908
2023-01-11T22:21:33.3493978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3494156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3494539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3494735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3495103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3495277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3495654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3495843Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3496092Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3496323Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3496729Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3497133Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3497371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3497602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3498386Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T22:21:33.3498501Z   warnings.warn(
2023-01-11T22:21:33.3499333Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T22:21:33.3499486Z   warnings.warn(
2023-01-11T22:21:33.3499723Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3499933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3500169Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3500402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3500502Z ok (6.251s)
2023-01-11T22:21:33.3500521Z 
2023-01-11T22:21:33.3500793Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3500908Z Ran 1 test in 6.251s
2023-01-11T22:21:33.3500928Z 
2023-01-11T22:21:33.3501021Z OK
2023-01-11T22:21:33.3501040Z 
2023-01-11T22:21:33.3501164Z Generating XML reports...
2023-01-11T22:21:33.3501603Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220425.xml
2023-01-11T22:21:33.3501979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3502157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3502543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3502739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3502758Z 
2023-01-11T22:21:33.3502865Z Running tests...
2023-01-11T22:21:33.3503130Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3503448Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3503745Z   test_ddp_apply_optim_in_backward_ignored_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3503951Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38022
2023-01-11T22:21:33.3504173Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38023
2023-01-11T22:21:33.3504544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3504720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3505100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3505290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3505663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3505839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3506215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3506386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3506634Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3507037Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3507276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3507670Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3507953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3508189Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3509023Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T22:21:33.3509136Z   warnings.warn(
2023-01-11T22:21:33.3509920Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:738: UserWarning: DDP + apply_optim_in_backward will currently set all parameter gradients to None. If this is not the desired behavior, please set env variable DDP_OVERLAPPED_OPTIM_SET_GRADS_TO_NONE=0, and manually setgradients to None/zero as desired.
2023-01-11T22:21:33.3510013Z   warnings.warn(
2023-01-11T22:21:33.3510118Z ok (6.135s)
2023-01-11T22:21:33.3510138Z 
2023-01-11T22:21:33.3510455Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3510567Z Ran 1 test in 6.135s
2023-01-11T22:21:33.3510590Z 
2023-01-11T22:21:33.3510684Z OK
2023-01-11T22:21:33.3510703Z 
2023-01-11T22:21:33.3510826Z Generating XML reports...
2023-01-11T22:21:33.3511277Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220434.xml
2023-01-11T22:21:33.3511649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3511809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3512189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3512382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3512401Z 
2023-01-11T22:21:33.3512513Z Running tests...
2023-01-11T22:21:33.3512776Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3513091Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3513362Z   test_ddp_broadcast_buffer (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3513582Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38139
2023-01-11T22:21:33.3513801Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38140
2023-01-11T22:21:33.3514155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3514333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3514713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3514908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3515273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3515448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3515827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3516018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3516246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3516488Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3516889Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3517355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3517595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3517871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3518109Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3518347Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3518449Z ok (5.535s)
2023-01-11T22:21:33.3518469Z 
2023-01-11T22:21:33.3518719Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3518834Z Ran 1 test in 5.535s
2023-01-11T22:21:33.3518853Z 
2023-01-11T22:21:33.3518947Z OK
2023-01-11T22:21:33.3518966Z 
2023-01-11T22:21:33.3519090Z Generating XML reports...
2023-01-11T22:21:33.3519550Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220443.xml
2023-01-11T22:21:33.3519923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3520104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3520487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3520662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3520699Z 
2023-01-11T22:21:33.3520788Z Running tests...
2023-01-11T22:21:33.3521051Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3521367Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3521642Z   test_ddp_broadcast_buffer_via_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3521867Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38254
2023-01-11T22:21:33.3522086Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38255
2023-01-11T22:21:33.3522460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3522639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3523001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3523191Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3523562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3523734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3524117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3524481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3524743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3524990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3525383Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3525781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3526013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3526245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3526556Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3526802Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3527087Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3527323Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3527424Z ok (5.518s)
2023-01-11T22:21:33.3527445Z 
2023-01-11T22:21:33.3527696Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3527806Z Ran 1 test in 5.518s
2023-01-11T22:21:33.3527825Z 
2023-01-11T22:21:33.3527919Z OK
2023-01-11T22:21:33.3527938Z 
2023-01-11T22:21:33.3528062Z Generating XML reports...
2023-01-11T22:21:33.3528521Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220451.xml
2023-01-11T22:21:33.3528900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3529078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3529459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3529655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3529675Z 
2023-01-11T22:21:33.3529765Z Running tests...
2023-01-11T22:21:33.3530030Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3530346Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3530620Z   test_ddp_buffer_hook_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3531377Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78641 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.629s)
2023-01-11T22:21:33.3531398Z 
2023-01-11T22:21:33.3531667Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3531781Z Ran 1 test in 1.629s
2023-01-11T22:21:33.3531801Z 
2023-01-11T22:21:33.3531908Z OK (skipped=1)
2023-01-11T22:21:33.3531927Z 
2023-01-11T22:21:33.3532050Z Generating XML reports...
2023-01-11T22:21:33.3532482Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220459.xml
2023-01-11T22:21:33.3532855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3533030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3533410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3533605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3533625Z 
2023-01-11T22:21:33.3533733Z Running tests...
2023-01-11T22:21:33.3533999Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3534312Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3534603Z   test_ddp_buffer_hook_allreduce_return_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3535336Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77261 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.634s)
2023-01-11T22:21:33.3535375Z 
2023-01-11T22:21:33.3535618Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3535781Z Ran 1 test in 1.634s
2023-01-11T22:21:33.3535802Z 
2023-01-11T22:21:33.3535914Z OK (skipped=1)
2023-01-11T22:21:33.3535933Z 
2023-01-11T22:21:33.3536057Z Generating XML reports...
2023-01-11T22:21:33.3536557Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220503.xml
2023-01-11T22:21:33.3536931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3537108Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3537494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3537688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3537709Z 
2023-01-11T22:21:33.3537799Z Running tests...
2023-01-11T22:21:33.3538061Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3538377Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3538666Z   test_ddp_build_debug_param_to_name_mapping (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3538893Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38437
2023-01-11T22:21:33.3539115Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38438
2023-01-11T22:21:33.3539488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3539664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3540027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3540218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3540590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3540764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3541144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3541333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3541582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3541828Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3542232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3542614Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3542849Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3543080Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3543295Z <class 'torch.nn.parameter.Parameter'>
2023-01-11T22:21:33.3543396Z ok (5.043s)
2023-01-11T22:21:33.3543416Z 
2023-01-11T22:21:33.3543679Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3543790Z Ran 1 test in 5.044s
2023-01-11T22:21:33.3543809Z 
2023-01-11T22:21:33.3543900Z OK
2023-01-11T22:21:33.3543919Z 
2023-01-11T22:21:33.3544024Z Generating XML reports...
2023-01-11T22:21:33.3544478Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220507.xml
2023-01-11T22:21:33.3544850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3545080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3545475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3545719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3545739Z 
2023-01-11T22:21:33.3545847Z Running tests...
2023-01-11T22:21:33.3546111Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3546426Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3546719Z   test_ddp_build_debug_param_to_name_mapping_requires_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3546942Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38548
2023-01-11T22:21:33.3547161Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38549
2023-01-11T22:21:33.3547537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3547712Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3548097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3548292Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3548659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3548814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3549191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3549382Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3549633Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3549880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3550286Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3550690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3550923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3551155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3551240Z ok (5.037s)
2023-01-11T22:21:33.3551259Z 
2023-01-11T22:21:33.3551527Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3551639Z Ran 1 test in 5.037s
2023-01-11T22:21:33.3551658Z 
2023-01-11T22:21:33.3551750Z OK
2023-01-11T22:21:33.3551769Z 
2023-01-11T22:21:33.3551895Z Generating XML reports...
2023-01-11T22:21:33.3552349Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220515.xml
2023-01-11T22:21:33.3552727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3552904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3553266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3553458Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3553477Z 
2023-01-11T22:21:33.3553587Z Running tests...
2023-01-11T22:21:33.3553850Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3554164Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3554476Z   test_ddp_comm_hook_logging (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3554705Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38659
2023-01-11T22:21:33.3554967Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38660
2023-01-11T22:21:33.3555341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3555498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3555881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3556074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3556439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3556616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3556994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3557188Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3557436Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3557664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3558065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3558463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3558694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3558928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3559168Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3559405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3559511Z ok (5.543s)
2023-01-11T22:21:33.3559531Z 
2023-01-11T22:21:33.3559797Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3559891Z Ran 1 test in 5.544s
2023-01-11T22:21:33.3559911Z 
2023-01-11T22:21:33.3560003Z OK
2023-01-11T22:21:33.3560022Z 
2023-01-11T22:21:33.3560147Z Generating XML reports...
2023-01-11T22:21:33.3560604Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220523.xml
2023-01-11T22:21:33.3560976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3561158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3561537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3561732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3561752Z 
2023-01-11T22:21:33.3561860Z Running tests...
2023-01-11T22:21:33.3562104Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3562417Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3562705Z   test_ddp_control_flow_different_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3562926Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38774
2023-01-11T22:21:33.3563148Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38775
2023-01-11T22:21:33.3563565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3563750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3564133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3564529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3564906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3565081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3565466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3565655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3565904Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3566155Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3566558Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3566959Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3567173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3567401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3568197Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:21:33.3568308Z ok (5.518s)
2023-01-11T22:21:33.3568328Z 
2023-01-11T22:21:33.3568597Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3568708Z Ran 1 test in 5.518s
2023-01-11T22:21:33.3568728Z 
2023-01-11T22:21:33.3568819Z OK
2023-01-11T22:21:33.3568838Z 
2023-01-11T22:21:33.3568963Z Generating XML reports...
2023-01-11T22:21:33.3569417Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220531.xml
2023-01-11T22:21:33.3569788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3569946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3570334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3570527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3570549Z 
2023-01-11T22:21:33.3570658Z Running tests...
2023-01-11T22:21:33.3570920Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3571237Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3571523Z   test_ddp_control_flow_same_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3572270Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78235 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.620s)
2023-01-11T22:21:33.3572291Z 
2023-01-11T22:21:33.3572624Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3572744Z Ran 1 test in 1.620s
2023-01-11T22:21:33.3572764Z 
2023-01-11T22:21:33.3572853Z OK (skipped=1)
2023-01-11T22:21:33.3572921Z 
2023-01-11T22:21:33.3573050Z Generating XML reports...
2023-01-11T22:21:33.3573508Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220539.xml
2023-01-11T22:21:33.3573882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3574060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3574441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3574632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3574651Z 
2023-01-11T22:21:33.3574761Z Running tests...
2023-01-11T22:21:33.3575010Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3575329Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3575595Z   test_ddp_create_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3575818Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 38923
2023-01-11T22:21:33.3576038Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 38924
2023-01-11T22:21:33.3576411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3576587Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3576963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3577156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3577506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3577684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3578057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3578245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3578492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3578740Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3579144Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3579546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3579775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3579990Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3580896Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T22:21:33.3581852Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T22:21:33.3583038Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1134.)
2023-01-11T22:21:33.3583329Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2023-01-11T22:21:33.3584509Z /opt/conda/lib/python3.10/site-packages/torch/autograd/__init__.py:197: UserWarning: Using backward() with create_graph=True will create a reference cycle between the parameter and its gradient which can cause a memory leak. We recommend using autograd.grad when creating the graph to avoid this. If you have to use this function, make sure to reset the .grad fields of your parameters to None after use to break the cycle and avoid the leak. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/engine.cpp:1134.)
2023-01-11T22:21:33.3584744Z   Variable._execution_engine.run_backward(  # Calls into the C++ engine to run the backward pass
2023-01-11T22:21:33.3584986Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3585225Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3586117Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T22:21:33.3587007Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T22:21:33.3587895Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T22:21:33.3588774Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T22:21:33.3589657Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T22:21:33.3590527Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T22:21:33.3591441Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T22:21:33.3592457Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T22:21:33.3593351Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T22:21:33.3594228Z [W reducer.cpp:380] Using DistributedDataParallel with create_graph=True  is not well-supported. The higher-order gradient will  not be synchronized across ranks, and backpropagation  through all_reduce operations will not occur. If you require  DDP to work with higher-order gradients for your use case,  please ping https://github.com/pytorch/pytorch/issues/63929
2023-01-11T22:21:33.3594336Z ok (4.251s)
2023-01-11T22:21:33.3594356Z 
2023-01-11T22:21:33.3594625Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3594736Z Ran 1 test in 4.251s
2023-01-11T22:21:33.3594756Z 
2023-01-11T22:21:33.3594847Z OK
2023-01-11T22:21:33.3594866Z 
2023-01-11T22:21:33.3594990Z Generating XML reports...
2023-01-11T22:21:33.3595448Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220543.xml
2023-01-11T22:21:33.3595826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3595988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3596375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3596570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3596589Z 
2023-01-11T22:21:33.3596697Z Running tests...
2023-01-11T22:21:33.3596962Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3597279Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3597528Z   test_ddp_device (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3598275Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77324 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.638s)
2023-01-11T22:21:33.3598298Z 
2023-01-11T22:21:33.3598560Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3598654Z Ran 1 test in 1.638s
2023-01-11T22:21:33.3598692Z 
2023-01-11T22:21:33.3598781Z OK (skipped=1)
2023-01-11T22:21:33.3598800Z 
2023-01-11T22:21:33.3598924Z Generating XML reports...
2023-01-11T22:21:33.3599381Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220550.xml
2023-01-11T22:21:33.3599755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3599934Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3600367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3600570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3600630Z 
2023-01-11T22:21:33.3600741Z Running tests...
2023-01-11T22:21:33.3600989Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3601306Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3601580Z   test_ddp_forward_backward_hook (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3601801Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39070
2023-01-11T22:21:33.3602023Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39071
2023-01-11T22:21:33.3602397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3602576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3602958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3603135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3603503Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3603675Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3604054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3604404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3604661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3604909Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3605322Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3605722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3605941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3606173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3606969Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1331: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2023-01-11T22:21:33.3607306Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2023-01-11T22:21:33.3608097Z /opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py:1331: UserWarning: Using a non-full backward hook when the forward contains multiple autograd Nodes is deprecated and will be removed in future versions. This hook will be missing some grad_input. Please use register_full_backward_hook to get the documented behavior.
2023-01-11T22:21:33.3608432Z   warnings.warn("Using a non-full backward hook when the forward contains multiple autograd Nodes "
2023-01-11T22:21:33.3608535Z ok (5.536s)
2023-01-11T22:21:33.3608555Z 
2023-01-11T22:21:33.3608817Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3608928Z Ran 1 test in 5.536s
2023-01-11T22:21:33.3608948Z 
2023-01-11T22:21:33.3609040Z OK
2023-01-11T22:21:33.3609059Z 
2023-01-11T22:21:33.3609181Z Generating XML reports...
2023-01-11T22:21:33.3609696Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220554.xml
2023-01-11T22:21:33.3610084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3610363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3610749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3610940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3610959Z 
2023-01-11T22:21:33.3611067Z Running tests...
2023-01-11T22:21:33.3611329Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3611646Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3611905Z   test_ddp_grad_div_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3612659Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78685 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.636s)
2023-01-11T22:21:33.3612702Z 
2023-01-11T22:21:33.3612946Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3613056Z Ran 1 test in 1.636s
2023-01-11T22:21:33.3613076Z 
2023-01-11T22:21:33.3613184Z OK (skipped=1)
2023-01-11T22:21:33.3613203Z 
2023-01-11T22:21:33.3613326Z Generating XML reports...
2023-01-11T22:21:33.3613777Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220602.xml
2023-01-11T22:21:33.3614148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3614325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3614710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3614885Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3614908Z 
2023-01-11T22:21:33.3615018Z Running tests...
2023-01-11T22:21:33.3615280Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3615594Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3615869Z   test_ddp_hook_parity_allreduce (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3616618Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77293 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.604s)
2023-01-11T22:21:33.3616639Z 
2023-01-11T22:21:33.3616904Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3617016Z Ran 1 test in 1.604s
2023-01-11T22:21:33.3617035Z 
2023-01-11T22:21:33.3617145Z OK (skipped=1)
2023-01-11T22:21:33.3617164Z 
2023-01-11T22:21:33.3617269Z Generating XML reports...
2023-01-11T22:21:33.3617720Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220606.xml
2023-01-11T22:21:33.3618091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3618269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3618655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3618848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3618868Z 
2023-01-11T22:21:33.3618977Z Running tests...
2023-01-11T22:21:33.3619291Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3619619Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3619940Z   test_ddp_hook_parity_allreduce_process_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3620167Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39283
2023-01-11T22:21:33.3620385Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39284
2023-01-11T22:21:33.3620761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3620938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3621318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3621516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3621883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3622057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3622414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3622605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3622853Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3623103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3623508Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3623911Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3624145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3624379Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3624602Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.3624845Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.3625244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.3625642Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.3625881Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3626121Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3626360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3626595Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3626696Z ok (5.839s)
2023-01-11T22:21:33.3626716Z 
2023-01-11T22:21:33.3626966Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3627078Z Ran 1 test in 5.839s
2023-01-11T22:21:33.3627097Z 
2023-01-11T22:21:33.3627188Z OK
2023-01-11T22:21:33.3627208Z 
2023-01-11T22:21:33.3627330Z Generating XML reports...
2023-01-11T22:21:33.3627787Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220610.xml
2023-01-11T22:21:33.3628160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3628389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3628786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3629028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3629047Z 
2023-01-11T22:21:33.3629138Z Running tests...
2023-01-11T22:21:33.3629403Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3629715Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3629994Z   test_ddp_hook_parity_post_localSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3630216Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39404
2023-01-11T22:21:33.3630432Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39405
2023-01-11T22:21:33.3630807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3630981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3631365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3631555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3631901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3632072Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3632451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3632638Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3632886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3633130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3633533Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3633934Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3634162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3634376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3634656Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T22:21:33.3634931Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T22:21:33.3635169Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3635397Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3635623Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3635861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3636136Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T22:21:33.3636393Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T22:21:33.3636665Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T22:21:33.3636932Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 10 iterations
2023-01-11T22:21:33.3637214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3637456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3637733Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3637959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3638237Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T22:21:33.3638511Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Start to apply local SGD after 10 iterations.
2023-01-11T22:21:33.3638770Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2023-01-11T22:21:33.3639039Z INFO:torch.distributed.algorithms.ddp_comm_hooks.post_localSGD_hook:Local SGD will be started after 1000 iterations
2023-01-11T22:21:33.3639272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3639501Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3639735Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3639965Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3640065Z ok (6.338s)
2023-01-11T22:21:33.3640084Z 
2023-01-11T22:21:33.3640354Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3640467Z Ran 1 test in 6.338s
2023-01-11T22:21:33.3640486Z 
2023-01-11T22:21:33.3640560Z OK
2023-01-11T22:21:33.3640579Z 
2023-01-11T22:21:33.3640700Z Generating XML reports...
2023-01-11T22:21:33.3641158Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220618.xml
2023-01-11T22:21:33.3641531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3641707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3642094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3642286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3642306Z 
2023-01-11T22:21:33.3642411Z Running tests...
2023-01-11T22:21:33.3642655Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3642966Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3643236Z   test_ddp_hook_parity_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3643987Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77378 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.642s)
2023-01-11T22:21:33.3644011Z 
2023-01-11T22:21:33.3644504Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3644624Z Ran 1 test in 1.642s
2023-01-11T22:21:33.3644644Z 
2023-01-11T22:21:33.3644750Z OK (skipped=1)
2023-01-11T22:21:33.3644769Z 
2023-01-11T22:21:33.3644893Z Generating XML reports...
2023-01-11T22:21:33.3645352Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220627.xml
2023-01-11T22:21:33.3645723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3645880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3646352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3646556Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3646629Z 
2023-01-11T22:21:33.3646739Z Running tests...
2023-01-11T22:21:33.3647004Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3647316Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3647590Z   test_ddp_hook_pickling_powerSGD (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3647808Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39553
2023-01-11T22:21:33.3648010Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39554
2023-01-11T22:21:33.3648381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3648559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3648938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3649131Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3649496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3649669Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3650042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3650233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3650462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3650705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3651109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3651509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3651737Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3651965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3652518Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:21:33.3653069Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 4; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:21:33.3653309Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3653541Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3653913Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2023-01-11T22:21:33.3654333Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Start to apply PowerSGD after 4 iterations.
2023-01-11T22:21:33.3654829Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2023-01-11T22:21:33.3655437Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:A zero tensor of length 10 that represents local error is created.
2023-01-11T22:21:33.3656010Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2023-01-11T22:21:33.3656415Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Compression stats: iter 4, total before compression 10, total after compression 10, rate 1.0
2023-01-11T22:21:33.3656741Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2023-01-11T22:21:33.3657066Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:Allocating contiguous memory of length 0 for Ps, and of length 0 for Qs, respectively.
2023-01-11T22:21:33.3657302Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3657531Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3657635Z ok (5.654s)
2023-01-11T22:21:33.3657656Z 
2023-01-11T22:21:33.3657924Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3658039Z Ran 1 test in 5.654s
2023-01-11T22:21:33.3658059Z 
2023-01-11T22:21:33.3658150Z OK
2023-01-11T22:21:33.3658169Z 
2023-01-11T22:21:33.3658292Z Generating XML reports...
2023-01-11T22:21:33.3658755Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220632.xml
2023-01-11T22:21:33.3659132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3659313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3659697Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3659873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3659914Z 
2023-01-11T22:21:33.3660004Z Running tests...
2023-01-11T22:21:33.3660268Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3660584Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3660901Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3661120Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39668
2023-01-11T22:21:33.3661338Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39669
2023-01-11T22:21:33.3661707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3661884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3662249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3662437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3662801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3662977Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3663349Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3663534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3663801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3664052Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3664512Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3664902Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3665181Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3665410Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3665647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3665880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3666111Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3666344Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3666444Z ok (5.828s)
2023-01-11T22:21:33.3666463Z 
2023-01-11T22:21:33.3666717Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3666828Z Ran 1 test in 5.828s
2023-01-11T22:21:33.3666847Z 
2023-01-11T22:21:33.3666941Z OK
2023-01-11T22:21:33.3666960Z 
2023-01-11T22:21:33.3667085Z Generating XML reports...
2023-01-11T22:21:33.3667538Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220640.xml
2023-01-11T22:21:33.3667907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3668082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3668461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3668651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3668670Z 
2023-01-11T22:21:33.3668761Z Running tests...
2023-01-11T22:21:33.3669026Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3669339Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3669651Z   test_ddp_hook_with_optimizer_parity_adam_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3669872Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39813
2023-01-11T22:21:33.3670087Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39814
2023-01-11T22:21:33.3670457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3670632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3670991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3671182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3671547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3671719Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3672098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3672286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3672530Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3672774Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3673171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3673602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3673838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3674068Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3674345Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3674578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3674810Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3675037Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3675136Z ok (5.727s)
2023-01-11T22:21:33.3675155Z 
2023-01-11T22:21:33.3675409Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3675518Z Ran 1 test in 5.727s
2023-01-11T22:21:33.3675538Z 
2023-01-11T22:21:33.3675629Z OK
2023-01-11T22:21:33.3675652Z 
2023-01-11T22:21:33.3675777Z Generating XML reports...
2023-01-11T22:21:33.3676231Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220648.xml
2023-01-11T22:21:33.3676608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3676785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3677164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3677356Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3677376Z 
2023-01-11T22:21:33.3677466Z Running tests...
2023-01-11T22:21:33.3677725Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3678039Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3678418Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3678639Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 39958
2023-01-11T22:21:33.3678856Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 39959
2023-01-11T22:21:33.3679223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3679395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3679769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3679942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3680310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3680482Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3680860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3681051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3681297Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3681541Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3681941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3682320Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3682599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3682833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3683072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3683352Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3683578Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3683811Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3683912Z ok (5.916s)
2023-01-11T22:21:33.3683931Z 
2023-01-11T22:21:33.3684386Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3684492Z Ran 1 test in 5.916s
2023-01-11T22:21:33.3684511Z 
2023-01-11T22:21:33.3684603Z OK
2023-01-11T22:21:33.3684622Z 
2023-01-11T22:21:33.3684744Z Generating XML reports...
2023-01-11T22:21:33.3685214Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220656.xml
2023-01-11T22:21:33.3685583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3685761Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3686141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3686331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3686351Z 
2023-01-11T22:21:33.3686457Z Running tests...
2023-01-11T22:21:33.3686700Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3687010Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3687385Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3687604Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40103
2023-01-11T22:21:33.3687823Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40104
2023-01-11T22:21:33.3688193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3688365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3688745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3688917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3689283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3689456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3689827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3690017Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3690262Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3690506Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3690904Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3691300Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3691514Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3691825Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3692075Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3692360Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3692592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3692823Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3692924Z ok (5.827s)
2023-01-11T22:21:33.3692943Z 
2023-01-11T22:21:33.3693210Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3693303Z Ran 1 test in 5.827s
2023-01-11T22:21:33.3693338Z 
2023-01-11T22:21:33.3693412Z OK
2023-01-11T22:21:33.3693431Z 
2023-01-11T22:21:33.3693554Z Generating XML reports...
2023-01-11T22:21:33.3694015Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220705.xml
2023-01-11T22:21:33.3694385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3694564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3694943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3695134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3695154Z 
2023-01-11T22:21:33.3695260Z Running tests...
2023-01-11T22:21:33.3695506Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3695822Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3696193Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3696417Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40248
2023-01-11T22:21:33.3696635Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40249
2023-01-11T22:21:33.3697009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3697183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3697562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3697749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3698100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3698270Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3698643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3698832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3699079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3699321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3699724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3700120Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3700349Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3700561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3700846Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3701088Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3701367Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3701600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3701704Z ok (5.826s)
2023-01-11T22:21:33.3701724Z 
2023-01-11T22:21:33.3701991Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3702104Z Ran 1 test in 5.826s
2023-01-11T22:21:33.3702123Z 
2023-01-11T22:21:33.3702197Z OK
2023-01-11T22:21:33.3702216Z 
2023-01-11T22:21:33.3702338Z Generating XML reports...
2023-01-11T22:21:33.3702792Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220713.xml
2023-01-11T22:21:33.3703165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3703340Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3703716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3703910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3703929Z 
2023-01-11T22:21:33.3704035Z Running tests...
2023-01-11T22:21:33.3704281Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3704599Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3704972Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_False_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3705192Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40393
2023-01-11T22:21:33.3705410Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40394
2023-01-11T22:21:33.3705778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3705957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3706338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3706529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3706875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3707048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3707421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3707614Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3707862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3708108Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3708510Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3708905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3709134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3709345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3709580Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3709879Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3710114Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3710433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3710536Z ok (5.826s)
2023-01-11T22:21:33.3710556Z 
2023-01-11T22:21:33.3710821Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3710933Z Ran 1 test in 5.826s
2023-01-11T22:21:33.3710953Z 
2023-01-11T22:21:33.3711026Z OK
2023-01-11T22:21:33.3711061Z 
2023-01-11T22:21:33.3711166Z Generating XML reports...
2023-01-11T22:21:33.3711622Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220722.xml
2023-01-11T22:21:33.3711997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3712176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3712558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3712750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3712770Z 
2023-01-11T22:21:33.3712876Z Running tests...
2023-01-11T22:21:33.3713133Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3713431Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3713800Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3714021Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40538
2023-01-11T22:21:33.3714238Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40539
2023-01-11T22:21:33.3714607Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3714781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3715165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3715354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3715718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3715877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3716255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3716440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3716692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3716937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3717341Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3717730Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3717961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3718188Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3718408Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3718639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3718915Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3719151Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3719295Z ok (5.833s)
2023-01-11T22:21:33.3719314Z 
2023-01-11T22:21:33.3719583Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3719694Z Ran 1 test in 5.833s
2023-01-11T22:21:33.3719714Z 
2023-01-11T22:21:33.3719806Z OK
2023-01-11T22:21:33.3719825Z 
2023-01-11T22:21:33.3719929Z Generating XML reports...
2023-01-11T22:21:33.3720380Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220730.xml
2023-01-11T22:21:33.3720749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3720926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3721307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3721498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3721520Z 
2023-01-11T22:21:33.3721627Z Running tests...
2023-01-11T22:21:33.3721888Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3722184Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3722556Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_False_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3722776Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40683
2023-01-11T22:21:33.3722990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40684
2023-01-11T22:21:33.3723362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3723537Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3723917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3724110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3724724Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3724885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3725262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3725452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3725697Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3725945Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3726348Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3726750Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3726980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3727207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3727426Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3727657Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3727883Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3728191Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3728301Z ok (5.839s)
2023-01-11T22:21:33.3728321Z 
2023-01-11T22:21:33.3728640Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3728752Z Ran 1 test in 5.839s
2023-01-11T22:21:33.3728772Z 
2023-01-11T22:21:33.3728862Z OK
2023-01-11T22:21:33.3728881Z 
2023-01-11T22:21:33.3728986Z Generating XML reports...
2023-01-11T22:21:33.3729445Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220738.xml
2023-01-11T22:21:33.3729817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3729996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3730376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3730570Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3730589Z 
2023-01-11T22:21:33.3730696Z Running tests...
2023-01-11T22:21:33.3730959Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3731271Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3731624Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3731846Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40828
2023-01-11T22:21:33.3732062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40829
2023-01-11T22:21:33.3732431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3732608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3732989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3733182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3733550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3733722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3734076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3734264Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3734510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3734752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3735154Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3735551Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3735780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3736007Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3736227Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3736461Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3736689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3736919Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3737112Z ok (5.730s)
2023-01-11T22:21:33.3737133Z 
2023-01-11T22:21:33.3737404Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3737558Z Ran 1 test in 5.730s
2023-01-11T22:21:33.3737578Z 
2023-01-11T22:21:33.3737669Z OK
2023-01-11T22:21:33.3737689Z 
2023-01-11T22:21:33.3737811Z Generating XML reports...
2023-01-11T22:21:33.3738253Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220747.xml
2023-01-11T22:21:33.3738623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3738797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3739175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3739368Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3739392Z 
2023-01-11T22:21:33.3739497Z Running tests...
2023-01-11T22:21:33.3739758Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3740077Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3740431Z   test_ddp_hook_with_optimizer_parity_adamw_grad_as_bucket_view_True_static_graph_True_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3740650Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 40973
2023-01-11T22:21:33.3740867Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 40974
2023-01-11T22:21:33.3741237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3741412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3741794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3741984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3742355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3742528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3742884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3743073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3743320Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3743563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3743965Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3744361Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3744593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3744823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3745059Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3745272Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3745506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3745738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3745839Z ok (5.831s)
2023-01-11T22:21:33.3745859Z 
2023-01-11T22:21:33.3746174Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3746290Z Ran 1 test in 5.831s
2023-01-11T22:21:33.3746309Z 
2023-01-11T22:21:33.3746402Z OK
2023-01-11T22:21:33.3746459Z 
2023-01-11T22:21:33.3746587Z Generating XML reports...
2023-01-11T22:21:33.3747030Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220755.xml
2023-01-11T22:21:33.3747403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3747575Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3747953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3748145Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3748165Z 
2023-01-11T22:21:33.3748272Z Running tests...
2023-01-11T22:21:33.3748536Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3748851Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3749167Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_False (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3749369Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41118
2023-01-11T22:21:33.3749586Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41119
2023-01-11T22:21:33.3749956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3750130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3750506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3750700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3751066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3751243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3751600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3751787Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3752033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3752278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3752678Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3753075Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3753304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3753535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3753771Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3753988Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3754222Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3754453Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3754553Z ok (5.700s)
2023-01-11T22:21:33.3754573Z 
2023-01-11T22:21:33.3754837Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3754949Z Ran 1 test in 5.701s
2023-01-11T22:21:33.3754968Z 
2023-01-11T22:21:33.3755107Z OK
2023-01-11T22:21:33.3755128Z 
2023-01-11T22:21:33.3755253Z Generating XML reports...
2023-01-11T22:21:33.3755712Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220803.xml
2023-01-11T22:21:33.3756115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3756291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3756667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3756857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3756876Z 
2023-01-11T22:21:33.3756986Z Running tests...
2023-01-11T22:21:33.3757246Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3757562Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3757873Z   test_ddp_hook_with_optimizer_parity_sgd_optimize_subset_True (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3758078Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41263
2023-01-11T22:21:33.3758296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41264
2023-01-11T22:21:33.3758663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3758837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3759212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3759401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3759763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3759935Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3760303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3760478Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3760723Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3760966Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3761365Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3761759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3761992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3762218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3762452Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3762686Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3762902Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3763134Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3763238Z ok (5.806s)
2023-01-11T22:21:33.3763257Z 
2023-01-11T22:21:33.3763521Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3763631Z Ran 1 test in 5.806s
2023-01-11T22:21:33.3763650Z 
2023-01-11T22:21:33.3763741Z OK
2023-01-11T22:21:33.3763760Z 
2023-01-11T22:21:33.3763882Z Generating XML reports...
2023-01-11T22:21:33.3764592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220812.xml
2023-01-11T22:21:33.3764975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3765211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3765591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3765780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3765800Z 
2023-01-11T22:21:33.3765907Z Running tests...
2023-01-11T22:21:33.3766170Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3766481Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3766747Z   test_ddp_ignore_params_arg (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3767505Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77325 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.641s)
2023-01-11T22:21:33.3767530Z 
2023-01-11T22:21:33.3767790Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3767884Z Ran 1 test in 1.641s
2023-01-11T22:21:33.3767903Z 
2023-01-11T22:21:33.3768009Z OK (skipped=1)
2023-01-11T22:21:33.3768028Z 
2023-01-11T22:21:33.3768148Z Generating XML reports...
2023-01-11T22:21:33.3768599Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220820.xml
2023-01-11T22:21:33.3768971Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3769147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3769524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3769717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3769737Z 
2023-01-11T22:21:33.3769826Z Running tests...
2023-01-11T22:21:33.3770086Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3770400Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3770655Z   test_ddp_inference (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3770872Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41442
2023-01-11T22:21:33.3771089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41443
2023-01-11T22:21:33.3771464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3771637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3772021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3772196Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3772563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3772734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3773107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3773296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3773542Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3773847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3774259Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3774693Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3774921Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3775150Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3775250Z ok (5.737s)
2023-01-11T22:21:33.3775270Z 
2023-01-11T22:21:33.3775533Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3775644Z Ran 1 test in 5.737s
2023-01-11T22:21:33.3775663Z 
2023-01-11T22:21:33.3775755Z OK
2023-01-11T22:21:33.3775774Z 
2023-01-11T22:21:33.3775903Z Generating XML reports...
2023-01-11T22:21:33.3776353Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220824.xml
2023-01-11T22:21:33.3776710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3776884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3777266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3777462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3777481Z 
2023-01-11T22:21:33.3777588Z Running tests...
2023-01-11T22:21:33.3777849Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3778161Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3778436Z   test_ddp_join_model_equivalence (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3778637Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41553
2023-01-11T22:21:33.3778855Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41554
2023-01-11T22:21:33.3779224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3779400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3779782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3779972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3780336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3780508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3780884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3781055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3781303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3781549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3781947Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3782341Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3782570Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3782848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3783091Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3783317Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3783756Z /opt/conda/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp8u6sdoq_'>
2023-01-11T22:21:33.3783920Z   _warnings.warn(warn_message, ResourceWarning)
2023-01-11T22:21:33.3784318Z /opt/conda/lib/python3.10/tempfile.py:860: ResourceWarning: Implicitly cleaning up <TemporaryDirectory '/tmp/tmp3vdsme1z'>
2023-01-11T22:21:33.3784483Z   _warnings.warn(warn_message, ResourceWarning)
2023-01-11T22:21:33.3784583Z ok (5.530s)
2023-01-11T22:21:33.3784603Z 
2023-01-11T22:21:33.3784865Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3784977Z Ran 1 test in 5.531s
2023-01-11T22:21:33.3784996Z 
2023-01-11T22:21:33.3785088Z OK
2023-01-11T22:21:33.3785111Z 
2023-01-11T22:21:33.3785218Z Generating XML reports...
2023-01-11T22:21:33.3785669Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220832.xml
2023-01-11T22:21:33.3786040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3786216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3786595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3786785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3786805Z 
2023-01-11T22:21:33.3786912Z Running tests...
2023-01-11T22:21:33.3787173Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3787468Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3787737Z   test_ddp_logging_data_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3787954Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41668
2023-01-11T22:21:33.3788237Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41669
2023-01-11T22:21:33.3788649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3788861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3789277Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3789510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3789865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3790220Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3790647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3790877Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3791157Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3791467Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3791909Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3792346Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3792563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3792890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3793214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3793539Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3793675Z ok (4.234s)
2023-01-11T22:21:33.3793695Z 
2023-01-11T22:21:33.3793999Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3794148Z Ran 1 test in 4.235s
2023-01-11T22:21:33.3794168Z 
2023-01-11T22:21:33.3794303Z OK
2023-01-11T22:21:33.3794323Z 
2023-01-11T22:21:33.3794428Z Generating XML reports...
2023-01-11T22:21:33.3794922Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220840.xml
2023-01-11T22:21:33.3795327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3795580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3796001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3796231Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3796251Z 
2023-01-11T22:21:33.3796431Z Running tests...
2023-01-11T22:21:33.3796741Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3797096Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3797351Z   test_ddp_logging_data_gpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3797608Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41811
2023-01-11T22:21:33.3797859Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41812
2023-01-11T22:21:33.3798302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3798520Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3798940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3799178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3799586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3799745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3800160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3800381Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3800662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3800975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3801418Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3801894Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3802161Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3802427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3802647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3802920Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3803054Z ok (5.513s)
2023-01-11T22:21:33.3803074Z 
2023-01-11T22:21:33.3803430Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3803631Z Ran 1 test in 5.513s
2023-01-11T22:21:33.3803651Z 
2023-01-11T22:21:33.3803780Z OK
2023-01-11T22:21:33.3803839Z 
2023-01-11T22:21:33.3804001Z Generating XML reports...
2023-01-11T22:21:33.3804734Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220847.xml
2023-01-11T22:21:33.3805160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3805322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3805739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3805966Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3805985Z 
2023-01-11T22:21:33.3806137Z Running tests...
2023-01-11T22:21:33.3806488Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3806891Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3807219Z   test_ddp_model_diff_num_params_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3807475Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 41926
2023-01-11T22:21:33.3807678Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 41927
2023-01-11T22:21:33.3808084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3808293Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3808718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3808944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3809387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3809600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3810021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3810286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3810521Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3810799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3811247Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3811678Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3811944Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3812273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3812560Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.3812836Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.3813272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.3813654Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.3813942Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:21:33.3814305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:21:33.3814754Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:21:33.3815243Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:21:33.3815420Z ok (5.170s)
2023-01-11T22:21:33.3815441Z 
2023-01-11T22:21:33.3815743Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3815900Z Ran 1 test in 5.170s
2023-01-11T22:21:33.3815920Z 
2023-01-11T22:21:33.3815995Z OK
2023-01-11T22:21:33.3816066Z 
2023-01-11T22:21:33.3816175Z Generating XML reports...
2023-01-11T22:21:33.3816669Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220855.xml
2023-01-11T22:21:33.3817080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3817294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3817742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3818009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3818029Z 
2023-01-11T22:21:33.3818186Z Running tests...
2023-01-11T22:21:33.3818488Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3818793Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3819109Z   test_ddp_model_diff_shape_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3819362Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42049
2023-01-11T22:21:33.3819615Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42050
2023-01-11T22:21:33.3820024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3820237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3820707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3820938Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3821345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3821504Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3821913Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3822137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3822420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3822727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3823179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3823646Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3823918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3824131Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3824408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.3824684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.3825168Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.3825619Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.3825943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:21:33.3826217Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:21:33.3826690Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:21:33.3827127Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:21:33.3827212Z ok (15.126s)
2023-01-11T22:21:33.3827283Z 
2023-01-11T22:21:33.3827538Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3827687Z Ran 1 test in 15.126s
2023-01-11T22:21:33.3827707Z 
2023-01-11T22:21:33.3827843Z OK
2023-01-11T22:21:33.3827863Z 
2023-01-11T22:21:33.3828055Z Generating XML reports...
2023-01-11T22:21:33.3828551Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220903.xml
2023-01-11T22:21:33.3828960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3829206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3829632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3829809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3829880Z 
2023-01-11T22:21:33.3829972Z Running tests...
2023-01-11T22:21:33.3830278Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3830634Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3830980Z   test_ddp_multiple_nested_unused_params_err_ignore_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3831239Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42172
2023-01-11T22:21:33.3831491Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42173
2023-01-11T22:21:33.3831933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3832147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3832515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3832750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3833157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3833393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3833808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3834033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3834316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3834626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3835015Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3835459Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3835775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3836046Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3836222Z ok (5.748s)
2023-01-11T22:21:33.3836242Z 
2023-01-11T22:21:33.3836545Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3836693Z Ran 1 test in 5.748s
2023-01-11T22:21:33.3836712Z 
2023-01-11T22:21:33.3836849Z OK
2023-01-11T22:21:33.3836869Z 
2023-01-11T22:21:33.3837063Z Generating XML reports...
2023-01-11T22:21:33.3837512Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220921.xml
2023-01-11T22:21:33.3837922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3838132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3838588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3838816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3838839Z 
2023-01-11T22:21:33.3838989Z Running tests...
2023-01-11T22:21:33.3839298Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3839650Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3839925Z   test_ddp_multiple_nested_unused_params_error (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3840219Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42287
2023-01-11T22:21:33.3840472Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42288
2023-01-11T22:21:33.3840881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3841091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3841508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3841750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3852263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3852502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3852921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3853118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3853371Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3853627Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3854039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3854444Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3854678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3854912Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3855015Z ok (5.647s)
2023-01-11T22:21:33.3855037Z 
2023-01-11T22:21:33.3855288Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3855401Z Ran 1 test in 5.648s
2023-01-11T22:21:33.3855420Z 
2023-01-11T22:21:33.3855511Z OK
2023-01-11T22:21:33.3855531Z 
2023-01-11T22:21:33.3855653Z Generating XML reports...
2023-01-11T22:21:33.3856243Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220929.xml
2023-01-11T22:21:33.3856644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3856891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3857280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3857473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3857493Z 
2023-01-11T22:21:33.3857585Z Running tests...
2023-01-11T22:21:33.3857851Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3858166Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3858429Z   test_ddp_namedtuple (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3858652Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42402
2023-01-11T22:21:33.3858872Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42403
2023-01-11T22:21:33.3859245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3859421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3859788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3859979Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3860343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3860518Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3860898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3861090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3861342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3861586Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3861986Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3862369Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3862601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3862831Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3862933Z ok (5.511s)
2023-01-11T22:21:33.3862954Z 
2023-01-11T22:21:33.3863217Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3863329Z Ran 1 test in 5.511s
2023-01-11T22:21:33.3863351Z 
2023-01-11T22:21:33.3863443Z OK
2023-01-11T22:21:33.3863462Z 
2023-01-11T22:21:33.3863586Z Generating XML reports...
2023-01-11T22:21:33.3864024Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220937.xml
2023-01-11T22:21:33.3864397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3864573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3864957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3865149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3865168Z 
2023-01-11T22:21:33.3865330Z Running tests...
2023-01-11T22:21:33.3865605Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3865922Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3866231Z   test_ddp_new_tensor_in_fwd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3866435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42513
2023-01-11T22:21:33.3866656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42514
2023-01-11T22:21:33.3867031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3867208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3867587Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3867782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3868149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3868327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3868680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3868867Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3869113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3869357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3869763Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3870164Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3870397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3870633Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3871426Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:21:33.3872219Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:21:33.3872464Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3872699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3872800Z ok (5.615s)
2023-01-11T22:21:33.3872820Z 
2023-01-11T22:21:33.3873072Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3873185Z Ran 1 test in 5.615s
2023-01-11T22:21:33.3873205Z 
2023-01-11T22:21:33.3873294Z OK
2023-01-11T22:21:33.3873314Z 
2023-01-11T22:21:33.3873487Z Generating XML reports...
2023-01-11T22:21:33.3873956Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220945.xml
2023-01-11T22:21:33.3874381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3874563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3874945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3875120Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3875159Z 
2023-01-11T22:21:33.3875249Z Running tests...
2023-01-11T22:21:33.3875511Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3875827Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3876114Z   test_ddp_new_tensor_in_fwd_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3876870Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78338 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.595s)
2023-01-11T22:21:33.3876894Z 
2023-01-11T22:21:33.3877153Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3877265Z Ran 1 test in 1.595s
2023-01-11T22:21:33.3877284Z 
2023-01-11T22:21:33.3877392Z OK (skipped=1)
2023-01-11T22:21:33.3877411Z 
2023-01-11T22:21:33.3877535Z Generating XML reports...
2023-01-11T22:21:33.3877971Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220953.xml
2023-01-11T22:21:33.3878343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3878522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3878908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3879103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3879122Z 
2023-01-11T22:21:33.3879229Z Running tests...
2023-01-11T22:21:33.3879492Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3879809Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3880075Z   test_ddp_profiling_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3880825Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77342 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.646s)
2023-01-11T22:21:33.3880863Z 
2023-01-11T22:21:33.3881108Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3881222Z Ran 1 test in 1.646s
2023-01-11T22:21:33.3881241Z 
2023-01-11T22:21:33.3881348Z OK (skipped=1)
2023-01-11T22:21:33.3881367Z 
2023-01-11T22:21:33.3881490Z Generating XML reports...
2023-01-11T22:21:33.3881943Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220957.xml
2023-01-11T22:21:33.3882315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3882492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3882873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3883100Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3883140Z 
2023-01-11T22:21:33.3883234Z Running tests...
2023-01-11T22:21:33.3883542Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3883863Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3884144Z   test_ddp_profiling_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3884650Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42696
2023-01-11T22:21:33.3884879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42697
2023-01-11T22:21:33.3885264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3885441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3885810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3886001Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3886372Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3886550Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3886926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3887114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3887361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3887609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3887998Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3888398Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3888636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3888867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3889205Z STAGE:2023-01-11 22:10:07 42696:42696 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3889537Z STAGE:2023-01-11 22:10:07 42697:42697 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3889775Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3890013Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3890566Z STAGE:2023-01-11 22:10:07 42697:42697 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:10:07 42696:42696 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3890590Z 
2023-01-11T22:21:33.3890944Z STAGE:2023-01-11 22:10:07 42697:42697 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3891274Z STAGE:2023-01-11 22:10:07 42696:42696 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3892150Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:21:33.3893024Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:21:33.3893466Z STAGE:2023-01-11 22:10:07 42696:42696 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3893772Z STAGE:2023-01-11 22:10:07 42697:42697 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3894106Z STAGE:2023-01-11 22:10:07 42697:42697 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3894438Z STAGE:2023-01-11 22:10:07 42696:42696 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3894783Z STAGE:2023-01-11 22:10:07 42697:42697 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3895136Z STAGE:2023-01-11 22:10:07 42696:42696 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3895238Z ok (6.064s)
2023-01-11T22:21:33.3895259Z 
2023-01-11T22:21:33.3895523Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3895634Z Ran 1 test in 6.064s
2023-01-11T22:21:33.3895654Z 
2023-01-11T22:21:33.3895729Z OK
2023-01-11T22:21:33.3895765Z 
2023-01-11T22:21:33.3895873Z Generating XML reports...
2023-01-11T22:21:33.3896333Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221002.xml
2023-01-11T22:21:33.3896712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3896893Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3897279Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3897475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3897494Z 
2023-01-11T22:21:33.3897601Z Running tests...
2023-01-11T22:21:33.3897864Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3898166Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3898434Z   test_ddp_python_error_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3898656Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42815
2023-01-11T22:21:33.3898877Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42816
2023-01-11T22:21:33.3899251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3899429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3899813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3900007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3900359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3900539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3900922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3901112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3901408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3901661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3902109Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3902509Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3902742Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3902954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3903056Z ok (5.123s)
2023-01-11T22:21:33.3903076Z 
2023-01-11T22:21:33.3903342Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3903453Z Ran 1 test in 5.123s
2023-01-11T22:21:33.3903472Z 
2023-01-11T22:21:33.3903565Z OK
2023-01-11T22:21:33.3903584Z 
2023-01-11T22:21:33.3903710Z Generating XML reports...
2023-01-11T22:21:33.3904164Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221010.xml
2023-01-11T22:21:33.3904541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3904717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3905077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3905266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3905285Z 
2023-01-11T22:21:33.3905392Z Running tests...
2023-01-11T22:21:33.3905653Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3905969Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3906249Z   test_ddp_returns_tensor_with_no_grad (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3907001Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78595 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.599s)
2023-01-11T22:21:33.3907025Z 
2023-01-11T22:21:33.3907285Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3907397Z Ran 1 test in 1.599s
2023-01-11T22:21:33.3907416Z 
2023-01-11T22:21:33.3907504Z OK (skipped=1)
2023-01-11T22:21:33.3907542Z 
2023-01-11T22:21:33.3907648Z Generating XML reports...
2023-01-11T22:21:33.3908100Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221018.xml
2023-01-11T22:21:33.3908474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3908654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3909039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3909230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3909249Z 
2023-01-11T22:21:33.3909356Z Running tests...
2023-01-11T22:21:33.3909618Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3909917Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3910202Z   test_ddp_shared_grad_acc_unused_params (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3910464Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 42960
2023-01-11T22:21:33.3910754Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 42961
2023-01-11T22:21:33.3911140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3911363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3911749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3911944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3912291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3912467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3912842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3913033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3913283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3913531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3913936Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3914336Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3914569Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3914781Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3915704Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:21:33.3915821Z   warnings.warn(
2023-01-11T22:21:33.3916737Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:21:33.3916847Z   warnings.warn(
2023-01-11T22:21:33.3917086Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3917317Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3917419Z ok (5.550s)
2023-01-11T22:21:33.3917438Z 
2023-01-11T22:21:33.3917707Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3917821Z Ran 1 test in 5.550s
2023-01-11T22:21:33.3917842Z 
2023-01-11T22:21:33.3917919Z OK
2023-01-11T22:21:33.3917955Z 
2023-01-11T22:21:33.3918061Z Generating XML reports...
2023-01-11T22:21:33.3918517Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221022.xml
2023-01-11T22:21:33.3918890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3919067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3919448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3919640Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3919660Z 
2023-01-11T22:21:33.3919767Z Running tests...
2023-01-11T22:21:33.3920084Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3920391Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3920714Z   test_ddp_static_graph_nested_types (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3921467Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77625 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.630s)
2023-01-11T22:21:33.3921488Z 
2023-01-11T22:21:33.3921747Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3921857Z Ran 1 test in 1.630s
2023-01-11T22:21:33.3921877Z 
2023-01-11T22:21:33.3921984Z OK (skipped=1)
2023-01-11T22:21:33.3922003Z 
2023-01-11T22:21:33.3922124Z Generating XML reports...
2023-01-11T22:21:33.3922579Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221030.xml
2023-01-11T22:21:33.3922952Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3923132Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3923498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3923690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3923709Z 
2023-01-11T22:21:33.3923817Z Running tests...
2023-01-11T22:21:33.3924079Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3924635Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3924923Z   test_ddp_sync_bn_training_vs_eval (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3925150Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43109
2023-01-11T22:21:33.3925381Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43110
2023-01-11T22:21:33.3925743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3925922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3926303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3926495Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3926860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3927034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3927414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3927604Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3927857Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3928084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3928487Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3928884Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3929115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3929345Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3929758Z STAGE:2023-01-11 22:10:39 43109:43109 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3930096Z STAGE:2023-01-11 22:10:39 43110:43110 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3930396Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3930616Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:21:33.3931171Z STAGE:2023-01-11 22:10:39 43109:43109 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:10:39 43110:43110 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3931191Z 
2023-01-11T22:21:33.3931766Z STAGE:2023-01-11 22:10:39 43110:43110 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:10:39 43109:43109 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3931787Z 
2023-01-11T22:21:33.3932118Z STAGE:2023-01-11 22:10:39 43109:43109 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.3932454Z STAGE:2023-01-11 22:10:40 43109:43109 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.3932803Z STAGE:2023-01-11 22:10:40 43109:43109 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.3932904Z ok (6.120s)
2023-01-11T22:21:33.3932923Z 
2023-01-11T22:21:33.3933185Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3933296Z Ran 1 test in 6.121s
2023-01-11T22:21:33.3933316Z 
2023-01-11T22:21:33.3933408Z OK
2023-01-11T22:21:33.3933426Z 
2023-01-11T22:21:33.3933548Z Generating XML reports...
2023-01-11T22:21:33.3933983Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221034.xml
2023-01-11T22:21:33.3934361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3934538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3934921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3935118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3935137Z 
2023-01-11T22:21:33.3935245Z Running tests...
2023-01-11T22:21:33.3935511Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3935828Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3936080Z   test_ddp_sync_module_states (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3936306Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43228
2023-01-11T22:21:33.3936527Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43229
2023-01-11T22:21:33.3936900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3937079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3937458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3937648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3938019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3938192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3938558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3938749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3939045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3939299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3939748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3940148Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3940381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3940614Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3940697Z ok (5.050s)
2023-01-11T22:21:33.3940742Z 
2023-01-11T22:21:33.3940990Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3941101Z Ran 1 test in 5.050s
2023-01-11T22:21:33.3941120Z 
2023-01-11T22:21:33.3941215Z OK
2023-01-11T22:21:33.3941235Z 
2023-01-11T22:21:33.3941357Z Generating XML reports...
2023-01-11T22:21:33.3941812Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221043.xml
2023-01-11T22:21:33.3942188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3942367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3942748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3942923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3942942Z 
2023-01-11T22:21:33.3943052Z Running tests...
2023-01-11T22:21:33.3943311Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3943631Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3943906Z   test_ddp_uneven_input_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3944130Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43339
2023-01-11T22:21:33.3944346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43340
2023-01-11T22:21:33.3944716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3944872Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3945250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3945441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3945806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3945983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3946361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3946552Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3946795Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3947041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3947424Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3947822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3948054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3948334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3948443Z ok (5.011s)
2023-01-11T22:21:33.3948500Z 
2023-01-11T22:21:33.3948771Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3948883Z Ran 1 test in 5.012s
2023-01-11T22:21:33.3948903Z 
2023-01-11T22:21:33.3948997Z OK
2023-01-11T22:21:33.3949016Z 
2023-01-11T22:21:33.3949139Z Generating XML reports...
2023-01-11T22:21:33.3949578Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221050.xml
2023-01-11T22:21:33.3949949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3950125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3950509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3950703Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3950722Z 
2023-01-11T22:21:33.3950831Z Running tests...
2023-01-11T22:21:33.3951097Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3951413Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3951672Z   test_ddp_uneven_input_join_disable (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3952425Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78684 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.602s)
2023-01-11T22:21:33.3952463Z 
2023-01-11T22:21:33.3952708Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3952822Z Ran 1 test in 1.602s
2023-01-11T22:21:33.3952841Z 
2023-01-11T22:21:33.3952950Z OK (skipped=1)
2023-01-11T22:21:33.3952968Z 
2023-01-11T22:21:33.3953091Z Generating XML reports...
2023-01-11T22:21:33.3953551Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221058.xml
2023-01-11T22:21:33.3953927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3954104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3954485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3954662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3954681Z 
2023-01-11T22:21:33.3954788Z Running tests...
2023-01-11T22:21:33.3955049Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3955366Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3955628Z   test_ddp_uneven_inputs (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3956381Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/75648 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.602s)
2023-01-11T22:21:33.3956402Z 
2023-01-11T22:21:33.3956667Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3956777Z Ran 1 test in 1.603s
2023-01-11T22:21:33.3956796Z 
2023-01-11T22:21:33.3956902Z OK (skipped=1)
2023-01-11T22:21:33.3956922Z 
2023-01-11T22:21:33.3957027Z Generating XML reports...
2023-01-11T22:21:33.3957532Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221102.xml
2023-01-11T22:21:33.3957916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3958138Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3958521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3958715Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3958734Z 
2023-01-11T22:21:33.3958844Z Running tests...
2023-01-11T22:21:33.3959107Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3959423Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3959696Z   test_ddp_uneven_inputs_stop_iteration_sync_bn (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3960446Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78113 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.640s)
2023-01-11T22:21:33.3960486Z 
2023-01-11T22:21:33.3960728Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3961073Z Ran 1 test in 1.640s
2023-01-11T22:21:33.3961092Z 
2023-01-11T22:21:33.3961201Z OK (skipped=1)
2023-01-11T22:21:33.3961219Z 
2023-01-11T22:21:33.3961347Z Generating XML reports...
2023-01-11T22:21:33.3961799Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221106.xml
2023-01-11T22:21:33.3962172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3962357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3962739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3962918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3962937Z 
2023-01-11T22:21:33.3963044Z Running tests...
2023-01-11T22:21:33.3963306Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3963624Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3963923Z   test_ddp_unused_params_rebuild_buckets_exception (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3964144Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43552
2023-01-11T22:21:33.3964593Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43553
2023-01-11T22:21:33.3964990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3965149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3965531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3965727Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3966094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3966269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3966648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3966838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3967087Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3967414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3967814Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3968269Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3968501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3968731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3968833Z ok (5.551s)
2023-01-11T22:21:33.3968853Z 
2023-01-11T22:21:33.3969119Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3969231Z Ran 1 test in 5.551s
2023-01-11T22:21:33.3969251Z 
2023-01-11T22:21:33.3969344Z OK
2023-01-11T22:21:33.3969363Z 
2023-01-11T22:21:33.3969492Z Generating XML reports...
2023-01-11T22:21:33.3969927Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221110.xml
2023-01-11T22:21:33.3970308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3970486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3970865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3971057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3971077Z 
2023-01-11T22:21:33.3971185Z Running tests...
2023-01-11T22:21:33.3971448Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3971762Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3972018Z   test_ddp_zero_output_features (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3972237Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43667
2023-01-11T22:21:33.3972461Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43668
2023-01-11T22:21:33.3972836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3973012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3973389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3973580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3973948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3974123Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3974488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3974676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3974925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3975172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3975577Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3975977Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3976210Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3976502Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3976880Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2023-01-11T22:21:33.3977179Z   warnings.warn("Initializing zero-element tensors is a no-op")
2023-01-11T22:21:33.3977554Z /opt/conda/lib/python3.10/site-packages/torch/nn/init.py:405: UserWarning: Initializing zero-element tensors is a no-op
2023-01-11T22:21:33.3977807Z   warnings.warn("Initializing zero-element tensors is a no-op")
2023-01-11T22:21:33.3977909Z ok (5.072s)
2023-01-11T22:21:33.3977929Z 
2023-01-11T22:21:33.3978194Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3978306Z Ran 1 test in 5.072s
2023-01-11T22:21:33.3978325Z 
2023-01-11T22:21:33.3978417Z OK
2023-01-11T22:21:33.3978436Z 
2023-01-11T22:21:33.3978561Z Generating XML reports...
2023-01-11T22:21:33.3979008Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221119.xml
2023-01-11T22:21:33.3979379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3979558Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3979943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3980137Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3980156Z 
2023-01-11T22:21:33.3980265Z Running tests...
2023-01-11T22:21:33.3980527Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3980843Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3981087Z   test_destroy_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3981310Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43778
2023-01-11T22:21:33.3981530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43779
2023-01-11T22:21:33.3981903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3982082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3982466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3982659Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3983025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3983197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3983558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3983751Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3983998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3984246Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3984653Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3985049Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3985281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3985513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3985758Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.3986046Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.3986458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.3986905Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.3987007Z ok (4.254s)
2023-01-11T22:21:33.3987028Z 
2023-01-11T22:21:33.3987292Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3987404Z Ran 1 test in 4.254s
2023-01-11T22:21:33.3987424Z 
2023-01-11T22:21:33.3987516Z OK
2023-01-11T22:21:33.3987535Z 
2023-01-11T22:21:33.3987658Z Generating XML reports...
2023-01-11T22:21:33.3988096Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221126.xml
2023-01-11T22:21:33.3988472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3988652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3989033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3989228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3989248Z 
2023-01-11T22:21:33.3989354Z Running tests...
2023-01-11T22:21:33.3989616Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3989934Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3990187Z   test_destroy_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.3990391Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 43893
2023-01-11T22:21:33.3990614Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 43894
2023-01-11T22:21:33.3990990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3991172Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3991552Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3991743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3992109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3992285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3992644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3992834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3993084Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.3993330Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.3993738Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3994139Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.3994368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.3994597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.3994839Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.3995062Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.3995511Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.3995955Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.3996058Z ok (4.260s)
2023-01-11T22:21:33.3996078Z 
2023-01-11T22:21:33.3996342Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3996455Z Ran 1 test in 4.260s
2023-01-11T22:21:33.3996474Z 
2023-01-11T22:21:33.3996566Z OK
2023-01-11T22:21:33.3996585Z 
2023-01-11T22:21:33.3996710Z Generating XML reports...
2023-01-11T22:21:33.3997162Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221133.xml
2023-01-11T22:21:33.3997518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.3997699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.3998078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.3998274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.3998293Z 
2023-01-11T22:21:33.3998400Z Running tests...
2023-01-11T22:21:33.3998664Z ----------------------------------------------------------------------
2023-01-11T22:21:33.3998978Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.3999257Z   test_detect_ddp_is_actually_static (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4000013Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78767 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.625s)
2023-01-11T22:21:33.4000035Z 
2023-01-11T22:21:33.4000295Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4000393Z Ran 1 test in 1.625s
2023-01-11T22:21:33.4000413Z 
2023-01-11T22:21:33.4000521Z OK (skipped=1)
2023-01-11T22:21:33.4000541Z 
2023-01-11T22:21:33.4000665Z Generating XML reports...
2023-01-11T22:21:33.4001116Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221140.xml
2023-01-11T22:21:33.4001488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4001664Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4002044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4002238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4002258Z 
2023-01-11T22:21:33.4002347Z Running tests...
2023-01-11T22:21:33.4002612Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4002929Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4003208Z   test_different_graph_across_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4003957Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78748 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.631s)
2023-01-11T22:21:33.4003977Z 
2023-01-11T22:21:33.4004451Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4004572Z Ran 1 test in 1.631s
2023-01-11T22:21:33.4004592Z 
2023-01-11T22:21:33.4004776Z OK (skipped=1)
2023-01-11T22:21:33.4004798Z 
2023-01-11T22:21:33.4004928Z Generating XML reports...
2023-01-11T22:21:33.4005398Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221144.xml
2023-01-11T22:21:33.4005814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4005992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4006374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4006566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4006586Z 
2023-01-11T22:21:33.4006695Z Running tests...
2023-01-11T22:21:33.4006958Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4007279Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4007555Z   test_dump_DDP_relevant_env_vars (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4007761Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44076
2023-01-11T22:21:33.4007985Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44077
2023-01-11T22:21:33.4008358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4008533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4008912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4009103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4009473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4009651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4010028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4010203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4010492Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4010739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4011145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4011545Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4011777Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4012010Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4012115Z ok (4.249s)
2023-01-11T22:21:33.4012135Z 
2023-01-11T22:21:33.4012406Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4012501Z Ran 1 test in 4.249s
2023-01-11T22:21:33.4012521Z 
2023-01-11T22:21:33.4012613Z OK
2023-01-11T22:21:33.4012633Z 
2023-01-11T22:21:33.4012756Z Generating XML reports...
2023-01-11T22:21:33.4013210Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221148.xml
2023-01-11T22:21:33.4013583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4013757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4014139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4014383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4014404Z 
2023-01-11T22:21:33.4014498Z Running tests...
2023-01-11T22:21:33.4014808Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4015119Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4015360Z   test_gather (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4015582Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44185
2023-01-11T22:21:33.4015805Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44186
2023-01-11T22:21:33.4016176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4016354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4016722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4016915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4017284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4017457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4017831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4018023Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4018270Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4018518Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4018922Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4019302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4019539Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4019768Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4020314Z STAGE:2023-01-11 22:11:59 44186:44186 ActivityProfilerController.cpp:300] Completed Stage: Warm UpSTAGE:2023-01-11 22:11:59 44185:44185 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4020335Z 
2023-01-11T22:21:33.4020674Z STAGE:2023-01-11 22:11:59 44186:44186 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4021025Z STAGE:2023-01-11 22:11:59 44186:44186 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4021361Z STAGE:2023-01-11 22:11:59 44185:44185 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4021711Z STAGE:2023-01-11 22:11:59 44185:44185 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4022045Z STAGE:2023-01-11 22:11:59 44186:44186 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4022365Z STAGE:2023-01-11 22:11:59 44185:44185 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4022682Z STAGE:2023-01-11 22:11:59 44185:44185 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4023030Z STAGE:2023-01-11 22:11:59 44185:44185 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4023364Z STAGE:2023-01-11 22:11:59 44186:44186 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4023711Z STAGE:2023-01-11 22:11:59 44186:44186 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4023860Z ok (4.342s)
2023-01-11T22:21:33.4023881Z 
2023-01-11T22:21:33.4024153Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4024309Z Ran 1 test in 4.342s
2023-01-11T22:21:33.4024328Z 
2023-01-11T22:21:33.4024424Z OK
2023-01-11T22:21:33.4024443Z 
2023-01-11T22:21:33.4024568Z Generating XML reports...
2023-01-11T22:21:33.4025011Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221155.xml
2023-01-11T22:21:33.4025384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4025560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4025946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4026138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4026162Z 
2023-01-11T22:21:33.4026271Z Running tests...
2023-01-11T22:21:33.4026534Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4026854Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4027094Z   test_gather_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4027315Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44298
2023-01-11T22:21:33.4027535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44299
2023-01-11T22:21:33.4027909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4028084Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4028467Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4028662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4029031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4029206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4029563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4029754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4030002Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4030248Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4030652Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4031055Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4031288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4031523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4031606Z ok (4.243s)
2023-01-11T22:21:33.4031644Z 
2023-01-11T22:21:33.4031893Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4032003Z Ran 1 test in 4.243s
2023-01-11T22:21:33.4032023Z 
2023-01-11T22:21:33.4032114Z OK
2023-01-11T22:21:33.4032133Z 
2023-01-11T22:21:33.4032257Z Generating XML reports...
2023-01-11T22:21:33.4032714Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221202.xml
2023-01-11T22:21:33.4033140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4033322Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4033705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4033924Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4033943Z 
2023-01-11T22:21:33.4034053Z Running tests...
2023-01-11T22:21:33.4034314Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4034631Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4034886Z   test_gather_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2023-01-11T22:21:33.4034906Z 
2023-01-11T22:21:33.4035166Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4035279Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4035298Z 
2023-01-11T22:21:33.4035409Z OK (skipped=1)
2023-01-11T22:21:33.4035428Z 
2023-01-11T22:21:33.4035551Z Generating XML reports...
2023-01-11T22:21:33.4035989Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221209.xml
2023-01-11T22:21:33.4036365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4036542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4036926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4037118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4037138Z 
2023-01-11T22:21:33.4037244Z Running tests...
2023-01-11T22:21:33.4037515Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4037835Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4038078Z   test_gather_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4038301Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44440
2023-01-11T22:21:33.4038525Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44441
2023-01-11T22:21:33.4038900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4039077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4039460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4039653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4040022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4040199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4040558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4040750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4040997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4041243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4041648Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4042048Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4042278Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4042605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4042834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.4043120Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.4043525Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4043919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4044455Z STAGE:2023-01-11 22:12:15 44441:44441 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4044794Z STAGE:2023-01-11 22:12:15 44440:44440 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4045139Z STAGE:2023-01-11 22:12:15 44441:44441 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4045492Z STAGE:2023-01-11 22:12:15 44441:44441 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4045831Z STAGE:2023-01-11 22:12:15 44440:44440 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4046159Z STAGE:2023-01-11 22:12:15 44440:44440 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4046485Z STAGE:2023-01-11 22:12:15 44441:44441 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4046815Z STAGE:2023-01-11 22:12:15 44440:44440 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4047151Z STAGE:2023-01-11 22:12:15 44440:44440 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4047495Z STAGE:2023-01-11 22:12:15 44440:44440 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4047833Z STAGE:2023-01-11 22:12:15 44441:44441 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4048174Z STAGE:2023-01-11 22:12:15 44441:44441 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4048279Z ok (4.342s)
2023-01-11T22:21:33.4048300Z 
2023-01-11T22:21:33.4048565Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4048658Z Ran 1 test in 4.342s
2023-01-11T22:21:33.4048695Z 
2023-01-11T22:21:33.4048769Z OK
2023-01-11T22:21:33.4048788Z 
2023-01-11T22:21:33.4048911Z Generating XML reports...
2023-01-11T22:21:33.4049370Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221211.xml
2023-01-11T22:21:33.4049746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4049926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4050315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4050510Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4050532Z 
2023-01-11T22:21:33.4050641Z Running tests...
2023-01-11T22:21:33.4050884Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4051201Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4051454Z   test_gather_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4051676Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44559
2023-01-11T22:21:33.4051895Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44560
2023-01-11T22:21:33.4052268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4052521Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4052917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4053147Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4053520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4053695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4054074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4054263Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4054509Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4054757Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4055158Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4055559Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4055772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4056002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4056158Z skip: Skipped due to small world size. (4.237s)
2023-01-11T22:21:33.4056178Z 
2023-01-11T22:21:33.4056444Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4056553Z Ran 1 test in 4.238s
2023-01-11T22:21:33.4056573Z 
2023-01-11T22:21:33.4056679Z OK (skipped=1)
2023-01-11T22:21:33.4056698Z 
2023-01-11T22:21:33.4056821Z Generating XML reports...
2023-01-11T22:21:33.4057280Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221218.xml
2023-01-11T22:21:33.4057650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4057811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4058191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4058379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4058399Z 
2023-01-11T22:21:33.4058506Z Running tests...
2023-01-11T22:21:33.4058769Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4059081Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4059337Z   test_gather_object (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4059559Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44668
2023-01-11T22:21:33.4059761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44669
2023-01-11T22:21:33.4060135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4060309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4060690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4060881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4061245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4061421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4061850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4062042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4062312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4062557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4062964Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4063363Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4063594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4063822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4063925Z ok (4.251s)
2023-01-11T22:21:33.4063949Z 
2023-01-11T22:21:33.4064215Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4064308Z Ran 1 test in 4.251s
2023-01-11T22:21:33.4064347Z 
2023-01-11T22:21:33.4064423Z OK
2023-01-11T22:21:33.4064442Z 
2023-01-11T22:21:33.4064564Z Generating XML reports...
2023-01-11T22:21:33.4065022Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221225.xml
2023-01-11T22:21:33.4065390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4065622Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4066146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4066340Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4066360Z 
2023-01-11T22:21:33.4066475Z Running tests...
2023-01-11T22:21:33.4066726Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4067044Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4067320Z   test_gather_object_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4068181Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/82866 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.608s)
2023-01-11T22:21:33.4068204Z 
2023-01-11T22:21:33.4068476Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4068589Z Ran 1 test in 1.608s
2023-01-11T22:21:33.4068609Z 
2023-01-11T22:21:33.4068717Z OK (skipped=1)
2023-01-11T22:21:33.4068736Z 
2023-01-11T22:21:33.4068864Z Generating XML reports...
2023-01-11T22:21:33.4069318Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221231.xml
2023-01-11T22:21:33.4069693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4069853Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4070234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4070424Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4070443Z 
2023-01-11T22:21:33.4070548Z Running tests...
2023-01-11T22:21:33.4070811Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4071125Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4071439Z   test_get_backend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4071668Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44811
2023-01-11T22:21:33.4071908Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44812
2023-01-11T22:21:33.4072284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4072459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4072840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4073033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4073399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4073573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4073951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4074123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4074377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4074625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4075029Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4075426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4075655Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4075888Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4076127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.4076369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.4076753Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4077149Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4077250Z ok (4.243s)
2023-01-11T22:21:33.4077270Z 
2023-01-11T22:21:33.4077532Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4077643Z Ran 1 test in 4.243s
2023-01-11T22:21:33.4077662Z 
2023-01-11T22:21:33.4077754Z OK
2023-01-11T22:21:33.4077773Z 
2023-01-11T22:21:33.4077897Z Generating XML reports...
2023-01-11T22:21:33.4078354Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221236.xml
2023-01-11T22:21:33.4078728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4078890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4079271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4079463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4079482Z 
2023-01-11T22:21:33.4079589Z Running tests...
2023-01-11T22:21:33.4079849Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4080163Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4080414Z   test_get_future (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4080693Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 44926
2023-01-11T22:21:33.4080901Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 44927
2023-01-11T22:21:33.4081322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4081500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4081879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4082071Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4082437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4082614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4082992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4083182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4083410Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4083657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4084059Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4084632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4084867Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4085099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4085203Z ok (4.231s)
2023-01-11T22:21:33.4085223Z 
2023-01-11T22:21:33.4085497Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4085610Z Ran 1 test in 4.231s
2023-01-11T22:21:33.4085629Z 
2023-01-11T22:21:33.4085708Z OK
2023-01-11T22:21:33.4085727Z 
2023-01-11T22:21:33.4085850Z Generating XML reports...
2023-01-11T22:21:33.4086303Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221242.xml
2023-01-11T22:21:33.4086675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4086851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4087231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4087423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4087443Z 
2023-01-11T22:21:33.4087552Z Running tests...
2023-01-11T22:21:33.4087800Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4088119Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4088368Z   test_get_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4088589Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45035
2023-01-11T22:21:33.4088806Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45036
2023-01-11T22:21:33.4089175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4089350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4089734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4089923Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4090345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4090528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4090962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4091151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4091401Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4091647Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4092049Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4092453Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4092667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4092894Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4092998Z ok (4.231s)
2023-01-11T22:21:33.4093018Z 
2023-01-11T22:21:33.4093280Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4093389Z Ran 1 test in 4.232s
2023-01-11T22:21:33.4093409Z 
2023-01-11T22:21:33.4093500Z OK
2023-01-11T22:21:33.4093519Z 
2023-01-11T22:21:33.4093640Z Generating XML reports...
2023-01-11T22:21:33.4094093Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221249.xml
2023-01-11T22:21:33.4094465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4094624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4095011Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4095204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4095223Z 
2023-01-11T22:21:33.4095331Z Running tests...
2023-01-11T22:21:33.4095591Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4095908Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4096175Z   test_get_rank_size_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4096395Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45144
2023-01-11T22:21:33.4096596Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45145
2023-01-11T22:21:33.4096972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4097146Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4097526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4097720Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4098085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4098258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4098635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4098825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4099054Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4099351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4099762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4100207Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4100440Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4100670Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4100915Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.4101160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.4101560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4101939Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4102043Z ok (4.227s)
2023-01-11T22:21:33.4102062Z 
2023-01-11T22:21:33.4102324Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4102435Z Ran 1 test in 4.227s
2023-01-11T22:21:33.4102454Z 
2023-01-11T22:21:33.4102546Z OK
2023-01-11T22:21:33.4102565Z 
2023-01-11T22:21:33.4102687Z Generating XML reports...
2023-01-11T22:21:33.4103143Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221256.xml
2023-01-11T22:21:33.4103518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4103694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4104062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4104255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4104277Z 
2023-01-11T22:21:33.4104386Z Running tests...
2023-01-11T22:21:33.4104646Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4104963Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4105229Z   test_get_rank_size_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4105451Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45259
2023-01-11T22:21:33.4105668Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45260
2023-01-11T22:21:33.4106020Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4106199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4106578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4106771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4107135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4107308Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4107684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4107872Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4108101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4108348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4108811Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4109254Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4109488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4109716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4109955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.4110198Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.4110638Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4111042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4111128Z ok (4.255s)
2023-01-11T22:21:33.4111148Z 
2023-01-11T22:21:33.4111416Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4111527Z Ran 1 test in 4.255s
2023-01-11T22:21:33.4111547Z 
2023-01-11T22:21:33.4111639Z OK
2023-01-11T22:21:33.4111658Z 
2023-01-11T22:21:33.4111780Z Generating XML reports...
2023-01-11T22:21:33.4112235Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221303.xml
2023-01-11T22:21:33.4112608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4112785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4113148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4113342Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4113362Z 
2023-01-11T22:21:33.4113470Z Running tests...
2023-01-11T22:21:33.4113737Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4114053Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4114320Z   test_invalid_static_graph (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4114542Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45374
2023-01-11T22:21:33.4114759Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45375
2023-01-11T22:21:33.4115137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4115295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4115679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4115870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4116242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4116420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4116802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4116991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4117239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4117466Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4117921Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4118325Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4118603Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4118833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4118936Z ok (5.505s)
2023-01-11T22:21:33.4118956Z 
2023-01-11T22:21:33.4119223Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4119334Z Ran 1 test in 5.505s
2023-01-11T22:21:33.4119354Z 
2023-01-11T22:21:33.4119445Z OK
2023-01-11T22:21:33.4119464Z 
2023-01-11T22:21:33.4119568Z Generating XML reports...
2023-01-11T22:21:33.4120023Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221309.xml
2023-01-11T22:21:33.4120400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4120578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4120965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4121158Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4121177Z 
2023-01-11T22:21:33.4121286Z Running tests...
2023-01-11T22:21:33.4121553Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4121850Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4122092Z   test_irecv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4122310Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45489
2023-01-11T22:21:33.4122530Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45490
2023-01-11T22:21:33.4122902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4123081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4123462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4123652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4124018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4124173Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4124734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4124928Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4125179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4125427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4125833Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4126233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4126466Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4126696Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4126781Z ok (4.245s)
2023-01-11T22:21:33.4126801Z 
2023-01-11T22:21:33.4127065Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4127246Z Ran 1 test in 4.246s
2023-01-11T22:21:33.4127267Z 
2023-01-11T22:21:33.4127365Z OK
2023-01-11T22:21:33.4127385Z 
2023-01-11T22:21:33.4127513Z Generating XML reports...
2023-01-11T22:21:33.4128031Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221318.xml
2023-01-11T22:21:33.4128401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4128579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4128939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4129132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4129152Z 
2023-01-11T22:21:33.4129260Z Running tests...
2023-01-11T22:21:33.4129523Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4129842Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4130085Z   test_isend (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4130307Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45598
2023-01-11T22:21:33.4130523Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45599
2023-01-11T22:21:33.4130874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4131050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4131432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4131622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4131995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4132169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4132545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4132736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4132984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4133213Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4133617Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4134016Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4134248Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4134478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4134580Z ok (4.225s)
2023-01-11T22:21:33.4134602Z 
2023-01-11T22:21:33.4134867Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4134976Z Ran 1 test in 4.226s
2023-01-11T22:21:33.4134996Z 
2023-01-11T22:21:33.4135070Z OK
2023-01-11T22:21:33.4135106Z 
2023-01-11T22:21:33.4135212Z Generating XML reports...
2023-01-11T22:21:33.4135665Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221324.xml
2023-01-11T22:21:33.4136034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4136211Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4136642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4136840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4136860Z 
2023-01-11T22:21:33.4137012Z Running tests...
2023-01-11T22:21:33.4137280Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4137579Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4137853Z   test_isend_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4138074Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45707
2023-01-11T22:21:33.4138294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45708
2023-01-11T22:21:33.4138662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4138838Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4139220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4139412Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4139786Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4139941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4140312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4140500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4140746Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4140990Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4141394Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4141793Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4142028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4142239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4142576Z STAGE:2023-01-11 22:13:35 45708:45708 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4142902Z STAGE:2023-01-11 22:13:35 45707:45707 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4143240Z STAGE:2023-01-11 22:13:35 45708:45708 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4143596Z STAGE:2023-01-11 22:13:35 45708:45708 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4143931Z STAGE:2023-01-11 22:13:35 45707:45707 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4144275Z STAGE:2023-01-11 22:13:35 45707:45707 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4144379Z ok (4.342s)
2023-01-11T22:21:33.4144399Z 
2023-01-11T22:21:33.4144664Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4144757Z Ran 1 test in 4.342s
2023-01-11T22:21:33.4144777Z 
2023-01-11T22:21:33.4144869Z OK
2023-01-11T22:21:33.4144888Z 
2023-01-11T22:21:33.4145011Z Generating XML reports...
2023-01-11T22:21:33.4145467Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221331.xml
2023-01-11T22:21:33.4145837Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4146065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4146455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4146700Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4146719Z 
2023-01-11T22:21:33.4146828Z Running tests...
2023-01-11T22:21:33.4147076Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4147390Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4147660Z   test_isend_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4147882Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45820
2023-01-11T22:21:33.4148100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45821
2023-01-11T22:21:33.4148477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4148653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4149031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4149207Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4149574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4149746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4150120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4150307Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4150556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4150803Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4151203Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4151606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4151820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4152047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4152384Z STAGE:2023-01-11 22:13:42 45821:45821 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4152709Z STAGE:2023-01-11 22:13:42 45820:45820 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4153045Z STAGE:2023-01-11 22:13:42 45821:45821 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4153392Z STAGE:2023-01-11 22:13:42 45821:45821 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4153728Z STAGE:2023-01-11 22:13:42 45820:45820 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4154081Z STAGE:2023-01-11 22:13:42 45820:45820 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4154183Z ok (4.334s)
2023-01-11T22:21:33.4154203Z 
2023-01-11T22:21:33.4154450Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4154562Z Ran 1 test in 4.335s
2023-01-11T22:21:33.4154581Z 
2023-01-11T22:21:33.4154675Z OK
2023-01-11T22:21:33.4154694Z 
2023-01-11T22:21:33.4154818Z Generating XML reports...
2023-01-11T22:21:33.4155270Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221338.xml
2023-01-11T22:21:33.4155692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4155878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4156313Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4156485Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4156523Z 
2023-01-11T22:21:33.4156614Z Running tests...
2023-01-11T22:21:33.4156873Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4157186Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4157469Z   test_monitored_barrier_allreduce_hang (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4157692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 45933
2023-01-11T22:21:33.4157914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 45934
2023-01-11T22:21:33.4158287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4158465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4158829Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4159018Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4159384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4159559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4159935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4160124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4160374Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4160619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4161008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4161406Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4161635Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4161863Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4162102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.4162348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.4162739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4163137Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4163377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:21:33.4163600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:21:33.4163996Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:21:33.4164559Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:21:33.4164803Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T22:21:33.4164980Z ok (20.863s)
2023-01-11T22:21:33.4165001Z 
2023-01-11T22:21:33.4165287Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4165459Z Ran 1 test in 20.863s
2023-01-11T22:21:33.4165479Z 
2023-01-11T22:21:33.4165573Z OK
2023-01-11T22:21:33.4165593Z 
2023-01-11T22:21:33.4165717Z Generating XML reports...
2023-01-11T22:21:33.4166159Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221345.xml
2023-01-11T22:21:33.4166531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4166708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4167090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4167281Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4167304Z 
2023-01-11T22:21:33.4167411Z Running tests...
2023-01-11T22:21:33.4167674Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4167992Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4168295Z   test_monitored_barrier_allreduce_hang_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4168500Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46060
2023-01-11T22:21:33.4168717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46061
2023-01-11T22:21:33.4169090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4169267Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4169650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4169840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4170204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4170383Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4170742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4170931Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4171176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4171419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4171818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4172219Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4172453Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4172682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4172922Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.4173148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.4173547Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4173943Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4174250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:21:33.4174498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:21:33.4174941Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:21:33.4175332Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:21:33.4175570Z [E ProcessGroupGloo.cpp:2803] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T22:21:33.4175805Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Ranks 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T22:21:33.4175889Z ok (21.176s)
2023-01-11T22:21:33.4175909Z 
2023-01-11T22:21:33.4176177Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4176288Z Ran 1 test in 21.176s
2023-01-11T22:21:33.4176308Z 
2023-01-11T22:21:33.4176400Z OK
2023-01-11T22:21:33.4176423Z 
2023-01-11T22:21:33.4176547Z Generating XML reports...
2023-01-11T22:21:33.4177001Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221408.xml
2023-01-11T22:21:33.4177378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4177555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4177938Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4178113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4178133Z 
2023-01-11T22:21:33.4178240Z Running tests...
2023-01-11T22:21:33.4178502Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4178818Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4179105Z   test_monitored_barrier_failure_order (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4179325Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46187
2023-01-11T22:21:33.4179545Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46188
2023-01-11T22:21:33.4179915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4180073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4180454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4180644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4181012Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4181190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4181565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4181758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4182006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4182250Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4182636Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4183031Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4183259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4183535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4183698Z skip: Skipped due to small world size. (4.232s)
2023-01-11T22:21:33.4183795Z 
2023-01-11T22:21:33.4184070Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4184181Z Ran 1 test in 4.233s
2023-01-11T22:21:33.4184200Z 
2023-01-11T22:21:33.4184308Z OK (skipped=1)
2023-01-11T22:21:33.4184327Z 
2023-01-11T22:21:33.4184450Z Generating XML reports...
2023-01-11T22:21:33.4184887Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221432.xml
2023-01-11T22:21:33.4185260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4185437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4185820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4186010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4186030Z 
2023-01-11T22:21:33.4186141Z Running tests...
2023-01-11T22:21:33.4186403Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4186721Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4186972Z   test_monitored_barrier_gloo (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4187192Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46296
2023-01-11T22:21:33.4187408Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46297
2023-01-11T22:21:33.4187779Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4187954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4188338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4188528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4188896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4189068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4189425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4189612Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4189859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4190101Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4190507Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4190906Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4191141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4191369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4191581Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 2000 ms
2023-01-11T22:21:33.4191682Z ok (6.236s)
2023-01-11T22:21:33.4191702Z 
2023-01-11T22:21:33.4191965Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4192075Z Ran 1 test in 6.237s
2023-01-11T22:21:33.4192094Z 
2023-01-11T22:21:33.4192186Z OK
2023-01-11T22:21:33.4192206Z 
2023-01-11T22:21:33.4192382Z Generating XML reports...
2023-01-11T22:21:33.4192896Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221439.xml
2023-01-11T22:21:33.4193278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4193510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4193875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4194072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4194092Z 
2023-01-11T22:21:33.4194201Z Running tests...
2023-01-11T22:21:33.4194465Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4194777Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4195066Z   test_monitored_barrier_gloo_rank_0_timeout (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4195287Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46405
2023-01-11T22:21:33.4195509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46406
2023-01-11T22:21:33.4195863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4196039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4196418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4196610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4196974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4197149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4197531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4197721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4197969Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4198199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4198604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4199001Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4199231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4199461Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4199707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.4199955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.4200350Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4200744Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4200946Z [E ProcessGroupGloo.cpp:138] Rank 0 timed out in monitoredBarrier after 0 ms.
2023-01-11T22:21:33.4201124Z No ranks successfully processed in monitoredBarrier.
2023-01-11T22:21:33.4201351Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 0 ms
2023-01-11T22:21:33.4201452Z ok (4.255s)
2023-01-11T22:21:33.4201472Z 
2023-01-11T22:21:33.4201788Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4201904Z Ran 1 test in 4.255s
2023-01-11T22:21:33.4201923Z 
2023-01-11T22:21:33.4202014Z OK
2023-01-11T22:21:33.4202032Z 
2023-01-11T22:21:33.4202157Z Generating XML reports...
2023-01-11T22:21:33.4202638Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221448.xml
2023-01-11T22:21:33.4203010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4203188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4203569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4203761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4203780Z 
2023-01-11T22:21:33.4203888Z Running tests...
2023-01-11T22:21:33.4204150Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4204650Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4204935Z   test_monitored_barrier_gloo_subgroup (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4205142Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46520
2023-01-11T22:21:33.4205363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46521
2023-01-11T22:21:33.4205738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4205911Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4206292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4206482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4206853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4207029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4207388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4207575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4207819Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4208060Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4208456Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4208851Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4209084Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4209311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4209553Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.4209773Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.4210171Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4210600Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4210831Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 100 ms
2023-01-11T22:21:33.4210930Z ok (4.461s)
2023-01-11T22:21:33.4210951Z 
2023-01-11T22:21:33.4211290Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4211410Z Ran 1 test in 4.461s
2023-01-11T22:21:33.4211430Z 
2023-01-11T22:21:33.4211522Z OK
2023-01-11T22:21:33.4211592Z 
2023-01-11T22:21:33.4211722Z Generating XML reports...
2023-01-11T22:21:33.4212161Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221454.xml
2023-01-11T22:21:33.4212532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4212707Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4213087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4213278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4213298Z 
2023-01-11T22:21:33.4213404Z Running tests...
2023-01-11T22:21:33.4213668Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4213981Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4214245Z   test_monitored_barrier_wait_all_ranks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4214464Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 46635
2023-01-11T22:21:33.4214681Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 46636
2023-01-11T22:21:33.4215049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4215224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4215603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4215791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4216160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4216334Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4216704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4216892Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4217137Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4217382Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4217778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4218176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4218411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4218639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4218799Z skip: Skipped due to small world size. (4.241s)
2023-01-11T22:21:33.4218819Z 
2023-01-11T22:21:33.4219066Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4219176Z Ran 1 test in 4.241s
2023-01-11T22:21:33.4219196Z 
2023-01-11T22:21:33.4219300Z OK (skipped=1)
2023-01-11T22:21:33.4219319Z 
2023-01-11T22:21:33.4219448Z Generating XML reports...
2023-01-11T22:21:33.4219901Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221501.xml
2023-01-11T22:21:33.4220274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4220500Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4220892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4221111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4221150Z 
2023-01-11T22:21:33.4221239Z Running tests...
2023-01-11T22:21:33.4221503Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4221816Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4222223Z   test_nccl_backend_bool_allgather (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2023-01-11T22:21:33.4222243Z 
2023-01-11T22:21:33.4222502Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4222615Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4222634Z 
2023-01-11T22:21:33.4222740Z OK (skipped=1)
2023-01-11T22:21:33.4222763Z 
2023-01-11T22:21:33.4222885Z Generating XML reports...
2023-01-11T22:21:33.4223319Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221508.xml
2023-01-11T22:21:33.4223693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4223868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4224248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4224439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4224459Z 
2023-01-11T22:21:33.4224566Z Running tests...
2023-01-11T22:21:33.4224828Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4225139Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4225545Z   test_nccl_backend_bool_allreduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2023-01-11T22:21:33.4225566Z 
2023-01-11T22:21:33.4225811Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4225921Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4225940Z 
2023-01-11T22:21:33.4226043Z OK (skipped=1)
2023-01-11T22:21:33.4226062Z 
2023-01-11T22:21:33.4226183Z Generating XML reports...
2023-01-11T22:21:33.4226633Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221511.xml
2023-01-11T22:21:33.4227001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4227176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4227553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4227741Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4227761Z 
2023-01-11T22:21:33.4227852Z Running tests...
2023-01-11T22:21:33.4228116Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4228423Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4228824Z   test_nccl_backend_bool_broadcast (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.002s)
2023-01-11T22:21:33.4228844Z 
2023-01-11T22:21:33.4229098Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4229207Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4229226Z 
2023-01-11T22:21:33.4229329Z OK (skipped=1)
2023-01-11T22:21:33.4229348Z 
2023-01-11T22:21:33.4229467Z Generating XML reports...
2023-01-11T22:21:33.4229947Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221513.xml
2023-01-11T22:21:33.4230321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4230543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4230924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4231117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4231137Z 
2023-01-11T22:21:33.4231243Z Running tests...
2023-01-11T22:21:33.4231504Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4231820Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4232216Z   test_nccl_backend_bool_reduce (__main__.TestDistBackendWithSpawn) ... skip: Test requires backend to be one of {'nccl'} (0.003s)
2023-01-11T22:21:33.4232236Z 
2023-01-11T22:21:33.4232496Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4232590Z Ran 1 test in 0.003s
2023-01-11T22:21:33.4232610Z 
2023-01-11T22:21:33.4232720Z OK (skipped=1)
2023-01-11T22:21:33.4232739Z 
2023-01-11T22:21:33.4232859Z Generating XML reports...
2023-01-11T22:21:33.4233305Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221515.xml
2023-01-11T22:21:33.4233676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4233846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4234224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4234414Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4234434Z 
2023-01-11T22:21:33.4234523Z Running tests...
2023-01-11T22:21:33.4234785Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4235097Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4235401Z   test_nccl_high_priority_stream (__main__.TestDistBackendWithSpawn) ... skip: Only NCCL backend supports high priority stream (0.002s)
2023-01-11T22:21:33.4235421Z 
2023-01-11T22:21:33.4235677Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4235786Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4235806Z 
2023-01-11T22:21:33.4235908Z OK (skipped=1)
2023-01-11T22:21:33.4235927Z 
2023-01-11T22:21:33.4236053Z Generating XML reports...
2023-01-11T22:21:33.4236501Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221518.xml
2023-01-11T22:21:33.4236855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4237036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4237416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4237609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4237628Z 
2023-01-11T22:21:33.4237732Z Running tests...
2023-01-11T22:21:33.4237993Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4238302Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4238554Z   test_new_subgroups (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T22:21:33.4238575Z 
2023-01-11T22:21:33.4238826Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4238919Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4238939Z 
2023-01-11T22:21:33.4239042Z OK (skipped=1)
2023-01-11T22:21:33.4239121Z 
2023-01-11T22:21:33.4239247Z Generating XML reports...
2023-01-11T22:21:33.4239693Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221520.xml
2023-01-11T22:21:33.4240111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4240287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4240660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4240852Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4240871Z 
2023-01-11T22:21:33.4240960Z Running tests...
2023-01-11T22:21:33.4241221Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4241531Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4241799Z   test_new_subgroups_by_enumeration (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T22:21:33.4241822Z 
2023-01-11T22:21:33.4242079Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4242187Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4242206Z 
2023-01-11T22:21:33.4242312Z OK (skipped=1)
2023-01-11T22:21:33.4242331Z 
2023-01-11T22:21:33.4242451Z Generating XML reports...
2023-01-11T22:21:33.4242899Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221523.xml
2023-01-11T22:21:33.4243251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4243426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4243807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4243995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4244015Z 
2023-01-11T22:21:33.4244115Z Running tests...
2023-01-11T22:21:33.4244554Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4244883Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4245339Z   test_new_subgroups_by_enumeration_input_rank_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T22:21:33.4245360Z 
2023-01-11T22:21:33.4245629Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4245722Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4245741Z 
2023-01-11T22:21:33.4245846Z OK (skipped=1)
2023-01-11T22:21:33.4245865Z 
2023-01-11T22:21:33.4245988Z Generating XML reports...
2023-01-11T22:21:33.4246445Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221525.xml
2023-01-11T22:21:33.4246808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4246988Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4247370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4247560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4247580Z 
2023-01-11T22:21:33.4247687Z Running tests...
2023-01-11T22:21:33.4247926Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4248236Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4248534Z   test_new_subgroups_by_enumeration_negative_input_rank (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4248832Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47008
2023-01-11T22:21:33.4249061Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47009
2023-01-11T22:21:33.4249596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4249770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4250150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4250324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4250690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4250861Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4251241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4251427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4251677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4251921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4252327Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4252722Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4252937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4253162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4253258Z ok (4.162s)
2023-01-11T22:21:33.4253281Z 
2023-01-11T22:21:33.4253543Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4253654Z Ran 1 test in 4.162s
2023-01-11T22:21:33.4253676Z 
2023-01-11T22:21:33.4253767Z OK
2023-01-11T22:21:33.4253786Z 
2023-01-11T22:21:33.4253908Z Generating XML reports...
2023-01-11T22:21:33.4254362Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221527.xml
2023-01-11T22:21:33.4254716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4254891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4255271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4255462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4255481Z 
2023-01-11T22:21:33.4255588Z Running tests...
2023-01-11T22:21:33.4255851Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4256168Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4256461Z   test_new_subgroups_group_size_exceeds_world_size (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4256678Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47117
2023-01-11T22:21:33.4256878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47118
2023-01-11T22:21:33.4257251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4257424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4257803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4258043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4258418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4258639Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4259016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4259187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4259433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4259677Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4260079Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4260477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4260707Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4260936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4261036Z ok (4.156s)
2023-01-11T22:21:33.4261056Z 
2023-01-11T22:21:33.4261320Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4261413Z Ran 1 test in 4.156s
2023-01-11T22:21:33.4261433Z 
2023-01-11T22:21:33.4261525Z OK
2023-01-11T22:21:33.4261545Z 
2023-01-11T22:21:33.4261663Z Generating XML reports...
2023-01-11T22:21:33.4262109Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221534.xml
2023-01-11T22:21:33.4262478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4262657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4263035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4263227Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4263247Z 
2023-01-11T22:21:33.4263353Z Running tests...
2023-01-11T22:21:33.4263595Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4263907Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4264184Z   test_new_subgroups_overlap_not_allowed (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T22:21:33.4264203Z 
2023-01-11T22:21:33.4264457Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4264565Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4264584Z 
2023-01-11T22:21:33.4264688Z OK (skipped=1)
2023-01-11T22:21:33.4264711Z 
2023-01-11T22:21:33.4264831Z Generating XML reports...
2023-01-11T22:21:33.4265279Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221541.xml
2023-01-11T22:21:33.4265652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4265812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4266186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4266370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4266389Z 
2023-01-11T22:21:33.4266492Z Running tests...
2023-01-11T22:21:33.4266753Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4267065Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4267413Z   test_new_subgroups_world_size_not_divisible_by_group_size (__main__.TestDistBackendWithSpawn) ... skip: Test requires world size of 4 (0.002s)
2023-01-11T22:21:33.4267470Z 
2023-01-11T22:21:33.4267738Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4267831Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4267865Z 
2023-01-11T22:21:33.4267955Z OK (skipped=1)
2023-01-11T22:21:33.4267974Z 
2023-01-11T22:21:33.4268097Z Generating XML reports...
2023-01-11T22:21:33.4268547Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221543.xml
2023-01-11T22:21:33.4268914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4269089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4269471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4269660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4269681Z 
2023-01-11T22:21:33.4269791Z Running tests...
2023-01-11T22:21:33.4270034Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4270347Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4270629Z   test_output_unused_in_loss_dict_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4271383Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/78112 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.648s)
2023-01-11T22:21:33.4271404Z 
2023-01-11T22:21:33.4271667Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4271777Z Ran 1 test in 1.649s
2023-01-11T22:21:33.4271796Z 
2023-01-11T22:21:33.4271903Z OK (skipped=1)
2023-01-11T22:21:33.4271922Z 
2023-01-11T22:21:33.4272046Z Generating XML reports...
2023-01-11T22:21:33.4272495Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221546.xml
2023-01-11T22:21:33.4272863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4273021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4273395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4273581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4273601Z 
2023-01-11T22:21:33.4273707Z Running tests...
2023-01-11T22:21:33.4273971Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4274282Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4274565Z   test_output_unused_in_loss_tuple_module (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4274790Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47326
2023-01-11T22:21:33.4274990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47327
2023-01-11T22:21:33.4275365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4275540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4275920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4276111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4276528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4276705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4277134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4277325Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4277556Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4277960Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4278199Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4278596Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4278830Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4279059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4279164Z ok (5.525s)
2023-01-11T22:21:33.4279183Z 
2023-01-11T22:21:33.4279443Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4279535Z Ran 1 test in 5.526s
2023-01-11T22:21:33.4279571Z 
2023-01-11T22:21:33.4279645Z OK
2023-01-11T22:21:33.4279663Z 
2023-01-11T22:21:33.4279785Z Generating XML reports...
2023-01-11T22:21:33.4280240Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221550.xml
2023-01-11T22:21:33.4280609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4280785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4281169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4281359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4281382Z 
2023-01-11T22:21:33.4281489Z Running tests...
2023-01-11T22:21:33.4281736Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4282046Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4282316Z   test_periodic_model_averager (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4282534Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47441
2023-01-11T22:21:33.4282747Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47442
2023-01-11T22:21:33.4283121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4283297Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4283674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4283850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4284375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4284559Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4284946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4285135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4285383Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4285701Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4286116Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4286577Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4286791Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4287021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4287122Z ok (5.642s)
2023-01-11T22:21:33.4287142Z 
2023-01-11T22:21:33.4287402Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4287510Z Ran 1 test in 5.642s
2023-01-11T22:21:33.4287530Z 
2023-01-11T22:21:33.4287624Z OK
2023-01-11T22:21:33.4287643Z 
2023-01-11T22:21:33.4287765Z Generating XML reports...
2023-01-11T22:21:33.4288221Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221558.xml
2023-01-11T22:21:33.4288592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4288754Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4289138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4289328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4289347Z 
2023-01-11T22:21:33.4289452Z Running tests...
2023-01-11T22:21:33.4289711Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4290024Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4290315Z   test_periodic_model_averager_param_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4290534Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47553
2023-01-11T22:21:33.4290733Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47554
2023-01-11T22:21:33.4291100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4291275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4291651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4291844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4292202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4292377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4292759Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4292944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4293180Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4293585Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4293829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4294224Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4294451Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4294679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4294830Z ok (5.635s)
2023-01-11T22:21:33.4294851Z 
2023-01-11T22:21:33.4295119Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4295213Z Ran 1 test in 5.635s
2023-01-11T22:21:33.4295294Z 
2023-01-11T22:21:33.4295372Z OK
2023-01-11T22:21:33.4295391Z 
2023-01-11T22:21:33.4295516Z Generating XML reports...
2023-01-11T22:21:33.4295971Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221606.xml
2023-01-11T22:21:33.4296336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4296513Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4296892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4297083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4297103Z 
2023-01-11T22:21:33.4297211Z Running tests...
2023-01-11T22:21:33.4297457Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4297769Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4298051Z   test_post_localSGD_optimizer_parity (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4298798Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77123 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.607s)
2023-01-11T22:21:33.4298819Z 
2023-01-11T22:21:33.4299080Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4299190Z Ran 1 test in 1.607s
2023-01-11T22:21:33.4299209Z 
2023-01-11T22:21:33.4299314Z OK (skipped=1)
2023-01-11T22:21:33.4299333Z 
2023-01-11T22:21:33.4299455Z Generating XML reports...
2023-01-11T22:21:33.4299907Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221614.xml
2023-01-11T22:21:33.4300280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4300440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4300818Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4301005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4301024Z 
2023-01-11T22:21:33.4301131Z Running tests...
2023-01-11T22:21:33.4301390Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4301702Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4301999Z   test_post_localSGD_optimizer_parity_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4302747Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/77292 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.606s)
2023-01-11T22:21:33.4302771Z 
2023-01-11T22:21:33.4303028Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4303120Z Ran 1 test in 1.606s
2023-01-11T22:21:33.4303153Z 
2023-01-11T22:21:33.4303242Z OK (skipped=1)
2023-01-11T22:21:33.4303260Z 
2023-01-11T22:21:33.4303382Z Generating XML reports...
2023-01-11T22:21:33.4303836Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221618.xml
2023-01-11T22:21:33.4304268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4304448Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4304876Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4305067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4305087Z 
2023-01-11T22:21:33.4305193Z Running tests...
2023-01-11T22:21:33.4305437Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4305749Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4306063Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4306285Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47733
2023-01-11T22:21:33.4306509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47734
2023-01-11T22:21:33.4306883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4307059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4307436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4307622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4307969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4308141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4308520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4308711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4308955Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4309203Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4309603Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4309999Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4310213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4310485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4310637Z skip: Need at least 4 CUDA devices (4.111s)
2023-01-11T22:21:33.4310657Z 
2023-01-11T22:21:33.4310926Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4311035Z Ran 1 test in 4.111s
2023-01-11T22:21:33.4311055Z 
2023-01-11T22:21:33.4311158Z OK (skipped=1)
2023-01-11T22:21:33.4311177Z 
2023-01-11T22:21:33.4311303Z Generating XML reports...
2023-01-11T22:21:33.4311758Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221622.xml
2023-01-11T22:21:33.4312126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4312284Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4312657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4312847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4312866Z 
2023-01-11T22:21:33.4312972Z Running tests...
2023-01-11T22:21:33.4313282Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4313606Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4313933Z   test_post_localSGD_optimizer_parity_with_hierarchical_sgd_grad_is_view (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4314198Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47842
2023-01-11T22:21:33.4314417Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47843
2023-01-11T22:21:33.4314772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4314948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4315324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4315517Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4315881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4316059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4316428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4316618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4316848Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4317090Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4317491Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4317890Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4318120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4318353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4318501Z skip: Need at least 4 CUDA devices (4.242s)
2023-01-11T22:21:33.4318521Z 
2023-01-11T22:21:33.4318784Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4318890Z Ran 1 test in 4.242s
2023-01-11T22:21:33.4318910Z 
2023-01-11T22:21:33.4319000Z OK (skipped=1)
2023-01-11T22:21:33.4319018Z 
2023-01-11T22:21:33.4319141Z Generating XML reports...
2023-01-11T22:21:33.4319592Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221629.xml
2023-01-11T22:21:33.4319963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4320141Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4320517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4320710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4320729Z 
2023-01-11T22:21:33.4320833Z Running tests...
2023-01-11T22:21:33.4321093Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4321389Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4321675Z   test_post_localSGD_optimizer_step_reload (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4322472Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/84886 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.608s)
2023-01-11T22:21:33.4322494Z 
2023-01-11T22:21:33.4322755Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4322918Z Ran 1 test in 1.609s
2023-01-11T22:21:33.4322939Z 
2023-01-11T22:21:33.4323043Z OK (skipped=1)
2023-01-11T22:21:33.4323063Z 
2023-01-11T22:21:33.4323183Z Generating XML reports...
2023-01-11T22:21:33.4323634Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221636.xml
2023-01-11T22:21:33.4324005Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4324163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4324793Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4324990Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4325009Z 
2023-01-11T22:21:33.4325118Z Running tests...
2023-01-11T22:21:33.4325379Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4325697Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4325962Z   test_reduce_full_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4326182Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 47985
2023-01-11T22:21:33.4326398Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 47986
2023-01-11T22:21:33.4326757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4326928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4327308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4327499Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4327867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4328042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4328414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4328603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4328832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4329077Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4329479Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4329878Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4330108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4330334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4330572Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.4330812Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.4331204Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4331585Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4332032Z STAGE:2023-01-11 22:16:44 47985:47985 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4332369Z STAGE:2023-01-11 22:16:44 47986:47986 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4332978Z STAGE:2023-01-11 22:16:44 47986:47986 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:16:44 47985:47985 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4332999Z 
2023-01-11T22:21:33.4333571Z STAGE:2023-01-11 22:16:44 47986:47986 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:16:44 47985:47985 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4333591Z 
2023-01-11T22:21:33.4333917Z STAGE:2023-01-11 22:16:44 47986:47986 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4334237Z STAGE:2023-01-11 22:16:44 47985:47985 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4334783Z STAGE:2023-01-11 22:16:44 47985:47985 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:16:44 47986:47986 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4334807Z 
2023-01-11T22:21:33.4335153Z STAGE:2023-01-11 22:16:44 47986:47986 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4335502Z STAGE:2023-01-11 22:16:44 47985:47985 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4335604Z ok (4.237s)
2023-01-11T22:21:33.4335623Z 
2023-01-11T22:21:33.4335881Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4335976Z Ran 1 test in 4.237s
2023-01-11T22:21:33.4335995Z 
2023-01-11T22:21:33.4336086Z OK
2023-01-11T22:21:33.4336105Z 
2023-01-11T22:21:33.4336225Z Generating XML reports...
2023-01-11T22:21:33.4336681Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221640.xml
2023-01-11T22:21:33.4337055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4337234Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4337617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4337810Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4337830Z 
2023-01-11T22:21:33.4337920Z Running tests...
2023-01-11T22:21:33.4338183Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4338498Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4338765Z   test_reduce_full_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4338991Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48104
2023-01-11T22:21:33.4339207Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48105
2023-01-11T22:21:33.4339581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4339756Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4340136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4340309Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4340675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4340846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4341222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4341456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4341708Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4341997Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4342400Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4342778Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4343006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4343236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4343476Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.4343721Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.4344116Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4344452Z STAGE:2023-01-11 22:16:51 48105:48105 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4344848Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4345176Z STAGE:2023-01-11 22:16:51 48104:48104 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4345722Z STAGE:2023-01-11 22:16:51 48105:48105 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:16:51 48104:48104 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4345744Z 
2023-01-11T22:21:33.4346317Z STAGE:2023-01-11 22:16:51 48104:48104 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:16:51 48105:48105 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4346338Z 
2023-01-11T22:21:33.4346648Z STAGE:2023-01-11 22:16:51 48105:48105 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4346975Z STAGE:2023-01-11 22:16:51 48104:48104 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4347309Z STAGE:2023-01-11 22:16:51 48104:48104 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4347638Z STAGE:2023-01-11 22:16:51 48105:48105 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4347984Z STAGE:2023-01-11 22:16:51 48104:48104 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4348334Z STAGE:2023-01-11 22:16:51 48105:48105 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4348435Z ok (4.232s)
2023-01-11T22:21:33.4348458Z 
2023-01-11T22:21:33.4348721Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4348831Z Ran 1 test in 4.232s
2023-01-11T22:21:33.4348853Z 
2023-01-11T22:21:33.4348927Z OK
2023-01-11T22:21:33.4348946Z 
2023-01-11T22:21:33.4349068Z Generating XML reports...
2023-01-11T22:21:33.4349523Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221647.xml
2023-01-11T22:21:33.4349897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4350070Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4350454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4350641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4350661Z 
2023-01-11T22:21:33.4350817Z Running tests...
2023-01-11T22:21:33.4351070Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4351385Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4351698Z   test_reduce_full_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4351914Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48223
2023-01-11T22:21:33.4352133Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48224
2023-01-11T22:21:33.4352511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4352684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4353063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4353255Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4353606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4353779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4354148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4354335Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4354581Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4354821Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4355219Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4355620Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4355847Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4356062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4356294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.4356531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.4356924Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4357311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4357636Z STAGE:2023-01-11 22:16:57 48224:48224 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4357952Z STAGE:2023-01-11 22:16:57 48223:48223 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4358491Z STAGE:2023-01-11 22:16:57 48224:48224 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:16:57 48223:48223 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4358515Z 
2023-01-11T22:21:33.4359077Z STAGE:2023-01-11 22:16:57 48223:48223 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:16:57 48224:48224 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4359097Z 
2023-01-11T22:21:33.4359413Z STAGE:2023-01-11 22:16:57 48224:48224 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4359713Z STAGE:2023-01-11 22:16:57 48223:48223 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4360087Z STAGE:2023-01-11 22:16:57 48223:48223 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4360423Z STAGE:2023-01-11 22:16:57 48224:48224 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4360764Z STAGE:2023-01-11 22:16:57 48223:48223 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4361154Z STAGE:2023-01-11 22:16:57 48224:48224 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4361254Z ok (4.342s)
2023-01-11T22:21:33.4361273Z 
2023-01-11T22:21:33.4361534Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4361645Z Ran 1 test in 4.343s
2023-01-11T22:21:33.4361663Z 
2023-01-11T22:21:33.4361750Z OK
2023-01-11T22:21:33.4361769Z 
2023-01-11T22:21:33.4361875Z Generating XML reports...
2023-01-11T22:21:33.4362330Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221653.xml
2023-01-11T22:21:33.4362705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4362876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4363260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4363450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4363470Z 
2023-01-11T22:21:33.4363576Z Running tests...
2023-01-11T22:21:33.4363836Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4364134Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4364626Z   test_reduce_full_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4364854Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48342
2023-01-11T22:21:33.4365073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48343
2023-01-11T22:21:33.4365449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4365626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4366007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4366198Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4366563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4366718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4367086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4367269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4367519Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4367764Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4368165Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4368562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4368787Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4369009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4369234Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.4369558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.4369969Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4370424Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4370763Z STAGE:2023-01-11 22:17:04 48342:48342 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4371084Z STAGE:2023-01-11 22:17:04 48343:48343 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4371634Z STAGE:2023-01-11 22:17:04 48343:48343 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:17:04 48342:48342 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4371655Z 
2023-01-11T22:21:33.4372232Z STAGE:2023-01-11 22:17:04 48342:48342 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:17:04 48343:48343 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4372252Z 
2023-01-11T22:21:33.4372578Z STAGE:2023-01-11 22:17:04 48343:48343 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4372901Z STAGE:2023-01-11 22:17:04 48342:48342 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4373444Z STAGE:2023-01-11 22:17:04 48343:48343 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:17:04 48342:48342 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4373465Z 
2023-01-11T22:21:33.4373797Z STAGE:2023-01-11 22:17:04 48343:48343 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4374143Z STAGE:2023-01-11 22:17:04 48342:48342 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4374246Z ok (4.238s)
2023-01-11T22:21:33.4374264Z 
2023-01-11T22:21:33.4374532Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4374643Z Ran 1 test in 4.238s
2023-01-11T22:21:33.4374662Z 
2023-01-11T22:21:33.4374757Z OK
2023-01-11T22:21:33.4374776Z 
2023-01-11T22:21:33.4374903Z Generating XML reports...
2023-01-11T22:21:33.4375357Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221700.xml
2023-01-11T22:21:33.4375727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4375887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4376268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4376457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4376476Z 
2023-01-11T22:21:33.4376582Z Running tests...
2023-01-11T22:21:33.4376846Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4377162Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4377424Z   test_reduce_group_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4377649Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48461
2023-01-11T22:21:33.4377849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48462
2023-01-11T22:21:33.4378221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4378389Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4378766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4378955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4379371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4379551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4379978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4380167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4380399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4380643Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4381042Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4381436Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4381671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4381901Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4382061Z skip: Skipped due to small world size. (4.252s)
2023-01-11T22:21:33.4382081Z 
2023-01-11T22:21:33.4382342Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4382448Z Ran 1 test in 4.252s
2023-01-11T22:21:33.4382468Z 
2023-01-11T22:21:33.4382557Z OK (skipped=1)
2023-01-11T22:21:33.4382576Z 
2023-01-11T22:21:33.4382699Z Generating XML reports...
2023-01-11T22:21:33.4383150Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221707.xml
2023-01-11T22:21:33.4383518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4383694Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4384077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4384270Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4384290Z 
2023-01-11T22:21:33.4384396Z Running tests...
2023-01-11T22:21:33.4384639Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4384951Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4385209Z   test_reduce_group_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4385428Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48570
2023-01-11T22:21:33.4385644Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48571
2023-01-11T22:21:33.4386014Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4386181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4386563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4386755Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4387104Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4387277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4387650Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4387838Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4388082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4388376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4388792Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4389244Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4389475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4389686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4389842Z skip: Skipped due to small world size. (4.338s)
2023-01-11T22:21:33.4389862Z 
2023-01-11T22:21:33.4390125Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4390235Z Ran 1 test in 4.339s
2023-01-11T22:21:33.4390255Z 
2023-01-11T22:21:33.4390359Z OK (skipped=1)
2023-01-11T22:21:33.4390378Z 
2023-01-11T22:21:33.4390503Z Generating XML reports...
2023-01-11T22:21:33.4390956Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221714.xml
2023-01-11T22:21:33.4391328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4391486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4391864Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4392050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4392069Z 
2023-01-11T22:21:33.4392171Z Running tests...
2023-01-11T22:21:33.4392433Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4392743Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4393012Z   test_reduce_group_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4393235Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48679
2023-01-11T22:21:33.4393453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48680
2023-01-11T22:21:33.4393808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4393979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4394355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4394542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4394905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4395079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4395454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4395644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4395870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4396113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4396514Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4396908Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4397137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4397411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4397575Z skip: Skipped due to small world size. (4.138s)
2023-01-11T22:21:33.4397595Z 
2023-01-11T22:21:33.4397862Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4398013Z Ran 1 test in 4.138s
2023-01-11T22:21:33.4398033Z 
2023-01-11T22:21:33.4398121Z OK (skipped=1)
2023-01-11T22:21:33.4398140Z 
2023-01-11T22:21:33.4398262Z Generating XML reports...
2023-01-11T22:21:33.4398717Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221721.xml
2023-01-11T22:21:33.4399089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4399266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4399643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4399836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4399856Z 
2023-01-11T22:21:33.4399963Z Running tests...
2023-01-11T22:21:33.4400220Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4400521Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4400778Z   test_reduce_group_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4400995Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48788
2023-01-11T22:21:33.4401213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48789
2023-01-11T22:21:33.4401589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4401759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4402142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4402331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4402681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4402852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4403230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4403416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4403662Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4403908Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4404529Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4404941Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4405176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4405388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4405544Z skip: Skipped due to small world size. (4.237s)
2023-01-11T22:21:33.4405563Z 
2023-01-11T22:21:33.4405826Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4405929Z Ran 1 test in 4.237s
2023-01-11T22:21:33.4405949Z 
2023-01-11T22:21:33.4406050Z OK (skipped=1)
2023-01-11T22:21:33.4406069Z 
2023-01-11T22:21:33.4406189Z Generating XML reports...
2023-01-11T22:21:33.4406639Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221728.xml
2023-01-11T22:21:33.4407084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4407269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4407693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4407884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4407904Z 
2023-01-11T22:21:33.4408005Z Running tests...
2023-01-11T22:21:33.4408267Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4408582Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4408832Z   test_reduce_max (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4409047Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 48897
2023-01-11T22:21:33.4409266Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 48898
2023-01-11T22:21:33.4409619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4409797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4410175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4410364Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4410775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4410946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4411321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4411508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4411737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4411984Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4412386Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4412781Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4413011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4413239Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4413573Z STAGE:2023-01-11 22:17:38 48897:48897 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4413909Z STAGE:2023-01-11 22:17:38 48898:48898 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4414450Z STAGE:2023-01-11 22:17:38 48897:48897 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:17:38 48898:48898 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4414475Z 
2023-01-11T22:21:33.4415043Z STAGE:2023-01-11 22:17:38 48897:48897 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:17:38 48898:48898 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4415063Z 
2023-01-11T22:21:33.4415385Z STAGE:2023-01-11 22:17:38 48898:48898 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4415685Z STAGE:2023-01-11 22:17:38 48897:48897 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4416279Z STAGE:2023-01-11 22:17:38 48898:48898 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:17:38 48897:48897 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4416301Z 
2023-01-11T22:21:33.4416875Z STAGE:2023-01-11 22:17:38 48898:48898 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:17:38 48897:48897 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4416936Z 
2023-01-11T22:21:33.4417042Z ok (4.237s)
2023-01-11T22:21:33.4417062Z 
2023-01-11T22:21:33.4417327Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4417437Z Ran 1 test in 4.238s
2023-01-11T22:21:33.4417456Z 
2023-01-11T22:21:33.4417548Z OK
2023-01-11T22:21:33.4417567Z 
2023-01-11T22:21:33.4417691Z Generating XML reports...
2023-01-11T22:21:33.4418146Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221734.xml
2023-01-11T22:21:33.4418521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4418681Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4419068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4419258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4419278Z 
2023-01-11T22:21:33.4419382Z Running tests...
2023-01-11T22:21:33.4419639Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4419954Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4420198Z   test_reduce_min (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4420415Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49010
2023-01-11T22:21:33.4420628Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49011
2023-01-11T22:21:33.4420985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4421160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4421529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4421699Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4422078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4422269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4422645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4422830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4423061Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4423301Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4423701Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4424094Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4424318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4424548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4424882Z STAGE:2023-01-11 22:17:45 49011:49011 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4425203Z STAGE:2023-01-11 22:17:45 49010:49010 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4425798Z STAGE:2023-01-11 22:17:45 49010:49010 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:17:45 49011:49011 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4425857Z 
2023-01-11T22:21:33.4426435Z STAGE:2023-01-11 22:17:45 49010:49010 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:17:45 49011:49011 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4426456Z 
2023-01-11T22:21:33.4426783Z STAGE:2023-01-11 22:17:45 49011:49011 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4427085Z STAGE:2023-01-11 22:17:45 49010:49010 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4427416Z STAGE:2023-01-11 22:17:45 49010:49010 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4427753Z STAGE:2023-01-11 22:17:45 49011:49011 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4428099Z STAGE:2023-01-11 22:17:45 49010:49010 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4428445Z STAGE:2023-01-11 22:17:45 49011:49011 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4428550Z ok (4.238s)
2023-01-11T22:21:33.4428569Z 
2023-01-11T22:21:33.4428831Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4428943Z Ran 1 test in 4.238s
2023-01-11T22:21:33.4428962Z 
2023-01-11T22:21:33.4429037Z OK
2023-01-11T22:21:33.4429071Z 
2023-01-11T22:21:33.4429179Z Generating XML reports...
2023-01-11T22:21:33.4429631Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221741.xml
2023-01-11T22:21:33.4430002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4430180Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4430561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4430749Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4430769Z 
2023-01-11T22:21:33.4430873Z Running tests...
2023-01-11T22:21:33.4431135Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4431433Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4431714Z   test_reduce_multigpu (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl backend supports reduce multigpu (0.002s)
2023-01-11T22:21:33.4431734Z 
2023-01-11T22:21:33.4431993Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4432100Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4432119Z 
2023-01-11T22:21:33.4432226Z OK (skipped=1)
2023-01-11T22:21:33.4432245Z 
2023-01-11T22:21:33.4432369Z Generating XML reports...
2023-01-11T22:21:33.4432817Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221748.xml
2023-01-11T22:21:33.4433189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4433365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4433729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4433922Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4433941Z 
2023-01-11T22:21:33.4434048Z Running tests...
2023-01-11T22:21:33.4434309Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4434623Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4434937Z   test_reduce_product (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4435163Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49156
2023-01-11T22:21:33.4435429Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49157
2023-01-11T22:21:33.4435783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4435957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4436332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4436522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4436887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4437059Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4437432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4437622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4437870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4438096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4438499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4438898Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4439124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4439461Z STAGE:2023-01-11 22:17:54 49156:49156 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4439684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4440011Z STAGE:2023-01-11 22:17:54 49157:49157 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4440344Z STAGE:2023-01-11 22:17:54 49157:49157 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4440670Z STAGE:2023-01-11 22:17:54 49156:49156 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4440999Z STAGE:2023-01-11 22:17:54 49157:49157 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4441348Z STAGE:2023-01-11 22:17:54 49156:49156 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4441677Z STAGE:2023-01-11 22:17:54 49157:49157 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4442001Z STAGE:2023-01-11 22:17:54 49156:49156 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4442334Z STAGE:2023-01-11 22:17:54 49156:49156 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4442657Z STAGE:2023-01-11 22:17:54 49157:49157 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4442999Z STAGE:2023-01-11 22:17:54 49156:49156 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4443337Z STAGE:2023-01-11 22:17:54 49157:49157 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4443421Z ok (4.345s)
2023-01-11T22:21:33.4443457Z 
2023-01-11T22:21:33.4443706Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4443818Z Ran 1 test in 4.345s
2023-01-11T22:21:33.4443838Z 
2023-01-11T22:21:33.4443928Z OK
2023-01-11T22:21:33.4443948Z 
2023-01-11T22:21:33.4444069Z Generating XML reports...
2023-01-11T22:21:33.4444825Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221750.xml
2023-01-11T22:21:33.4445220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4445461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4445845Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4446019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4446056Z 
2023-01-11T22:21:33.4446148Z Running tests...
2023-01-11T22:21:33.4446409Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4446723Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4447019Z   test_reduce_scatter_tensor_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce_scatter_tensor (0.002s)
2023-01-11T22:21:33.4447039Z 
2023-01-11T22:21:33.4447297Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4447409Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4447431Z 
2023-01-11T22:21:33.4447539Z OK (skipped=1)
2023-01-11T22:21:33.4447558Z 
2023-01-11T22:21:33.4447679Z Generating XML reports...
2023-01-11T22:21:33.4448107Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221757.xml
2023-01-11T22:21:33.4448479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4448652Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4449033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4449221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4449241Z 
2023-01-11T22:21:33.4449351Z Running tests...
2023-01-11T22:21:33.4449613Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4449924Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4450185Z   test_reduce_scatter_v_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports reduce_scatter_v (0.003s)
2023-01-11T22:21:33.4450217Z 
2023-01-11T22:21:33.4450460Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4450569Z Ran 1 test in 0.003s
2023-01-11T22:21:33.4450589Z 
2023-01-11T22:21:33.4450693Z OK (skipped=1)
2023-01-11T22:21:33.4450711Z 
2023-01-11T22:21:33.4450834Z Generating XML reports...
2023-01-11T22:21:33.4451283Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221800.xml
2023-01-11T22:21:33.4451657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4451831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4452208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4452385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4452422Z 
2023-01-11T22:21:33.4452514Z Running tests...
2023-01-11T22:21:33.4452773Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4453092Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4453337Z   test_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4453556Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49335
2023-01-11T22:21:33.4453774Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49336
2023-01-11T22:21:33.4454195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4454377Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4454775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4454949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4455328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4455520Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4455895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4456085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4456335Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4456582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4456973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4457366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4457594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4457822Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4458156Z STAGE:2023-01-11 22:18:06 49335:49335 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4458479Z STAGE:2023-01-11 22:18:06 49336:49336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4459028Z STAGE:2023-01-11 22:18:06 49336:49336 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:18:06 49335:49335 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4459051Z 
2023-01-11T22:21:33.4459629Z STAGE:2023-01-11 22:18:06 49335:49335 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:18:06 49336:49336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4459649Z 
2023-01-11T22:21:33.4459982Z STAGE:2023-01-11 22:18:06 49336:49336 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4460303Z STAGE:2023-01-11 22:18:06 49335:49335 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4460638Z STAGE:2023-01-11 22:18:06 49336:49336 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4460950Z STAGE:2023-01-11 22:18:06 49335:49335 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4461292Z STAGE:2023-01-11 22:18:06 49336:49336 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4461640Z STAGE:2023-01-11 22:18:06 49335:49335 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4461743Z ok (4.258s)
2023-01-11T22:21:33.4461763Z 
2023-01-11T22:21:33.4462026Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4462136Z Ran 1 test in 4.258s
2023-01-11T22:21:33.4462155Z 
2023-01-11T22:21:33.4462246Z OK
2023-01-11T22:21:33.4462265Z 
2023-01-11T22:21:33.4462387Z Generating XML reports...
2023-01-11T22:21:33.4462823Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221802.xml
2023-01-11T22:21:33.4463195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4463420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4463803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4464041Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4464060Z 
2023-01-11T22:21:33.4464168Z Running tests...
2023-01-11T22:21:33.4464435Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4464753Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4465013Z   test_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2023-01-11T22:21:33.4465033Z 
2023-01-11T22:21:33.4465274Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4465387Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4465406Z 
2023-01-11T22:21:33.4465512Z OK (skipped=1)
2023-01-11T22:21:33.4465530Z 
2023-01-11T22:21:33.4465655Z Generating XML reports...
2023-01-11T22:21:33.4466108Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221809.xml
2023-01-11T22:21:33.4466484Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4466656Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4467036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4467230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4467250Z 
2023-01-11T22:21:33.4467340Z Running tests...
2023-01-11T22:21:33.4467602Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4467916Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4468182Z   test_reduce_sum_cuda_twice (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA reduce (0.002s)
2023-01-11T22:21:33.4468202Z 
2023-01-11T22:21:33.4468459Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4468568Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4468587Z 
2023-01-11T22:21:33.4468691Z OK (skipped=1)
2023-01-11T22:21:33.4468709Z 
2023-01-11T22:21:33.4468829Z Generating XML reports...
2023-01-11T22:21:33.4469277Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221811.xml
2023-01-11T22:21:33.4469635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4469810Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4470183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4470373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4470393Z 
2023-01-11T22:21:33.4470498Z Running tests...
2023-01-11T22:21:33.4470761Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4471075Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4471330Z   test_reduce_sum_twice (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4471531Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49514
2023-01-11T22:21:33.4471746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49515
2023-01-11T22:21:33.4472118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4472290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4472715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4472911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4473278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4473532Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4473917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4474089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4474333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4474582Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4474987Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4475389Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4475625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4475961Z STAGE:2023-01-11 22:18:24 49514:49514 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4476187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4476514Z STAGE:2023-01-11 22:18:24 49515:49515 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4477047Z STAGE:2023-01-11 22:18:24 49514:49514 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:18:24 49515:49515 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4477082Z 
2023-01-11T22:21:33.4477640Z STAGE:2023-01-11 22:18:24 49514:49514 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:18:24 49515:49515 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4477660Z 
2023-01-11T22:21:33.4477990Z STAGE:2023-01-11 22:18:24 49515:49515 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4478311Z STAGE:2023-01-11 22:18:24 49514:49514 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4478645Z STAGE:2023-01-11 22:18:24 49514:49514 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4478976Z STAGE:2023-01-11 22:18:24 49515:49515 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4479318Z STAGE:2023-01-11 22:18:24 49514:49514 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4479658Z STAGE:2023-01-11 22:18:24 49515:49515 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4479759Z ok (11.976s)
2023-01-11T22:21:33.4479782Z 
2023-01-11T22:21:33.4480047Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4480143Z Ran 1 test in 11.976s
2023-01-11T22:21:33.4480165Z 
2023-01-11T22:21:33.4480255Z OK
2023-01-11T22:21:33.4480274Z 
2023-01-11T22:21:33.4480397Z Generating XML reports...
2023-01-11T22:21:33.4480850Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221814.xml
2023-01-11T22:21:33.4481221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4481398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4481780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4481969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4481988Z 
2023-01-11T22:21:33.4482142Z Running tests...
2023-01-11T22:21:33.4482398Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4482710Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4482994Z   test_scatter (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4483212Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49627
2023-01-11T22:21:33.4483432Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49628
2023-01-11T22:21:33.4483808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4483981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4484588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4484774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4485154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4485331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4485711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4485899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4486144Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4486385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4486783Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4487182Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4487397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4487627Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4487961Z STAGE:2023-01-11 22:18:35 49627:49627 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4488282Z STAGE:2023-01-11 22:18:35 49628:49628 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4488826Z STAGE:2023-01-11 22:18:35 49628:49628 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:18:35 49627:49627 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4488847Z 
2023-01-11T22:21:33.4489192Z STAGE:2023-01-11 22:18:35 49628:49628 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4489542Z STAGE:2023-01-11 22:18:35 49627:49627 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4489870Z STAGE:2023-01-11 22:18:35 49627:49627 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4490189Z STAGE:2023-01-11 22:18:35 49628:49628 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4490505Z STAGE:2023-01-11 22:18:35 49628:49628 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4491057Z STAGE:2023-01-11 22:18:35 49627:49627 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:18:35 49628:49628 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4491077Z 
2023-01-11T22:21:33.4491418Z STAGE:2023-01-11 22:18:35 49627:49627 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4491516Z ok (4.261s)
2023-01-11T22:21:33.4491535Z 
2023-01-11T22:21:33.4491869Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4491988Z Ran 1 test in 4.261s
2023-01-11T22:21:33.4492007Z 
2023-01-11T22:21:33.4492099Z OK
2023-01-11T22:21:33.4492118Z 
2023-01-11T22:21:33.4492285Z Generating XML reports...
2023-01-11T22:21:33.4492812Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221831.xml
2023-01-11T22:21:33.4493184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4493343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4493722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4493911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4493931Z 
2023-01-11T22:21:33.4494038Z Running tests...
2023-01-11T22:21:33.4494300Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4494615Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4494870Z   test_scatter_checks (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4495092Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49740
2023-01-11T22:21:33.4495294Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49741
2023-01-11T22:21:33.4495671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4495840Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4496213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4496402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4496769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4496940Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4497323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4497513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4497744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4497989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4498391Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4498795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4499030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4499260Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4499363Z ok (4.152s)
2023-01-11T22:21:33.4499383Z 
2023-01-11T22:21:33.4499646Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4499739Z Ran 1 test in 4.152s
2023-01-11T22:21:33.4499772Z 
2023-01-11T22:21:33.4499847Z OK
2023-01-11T22:21:33.4499866Z 
2023-01-11T22:21:33.4499987Z Generating XML reports...
2023-01-11T22:21:33.4500441Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221838.xml
2023-01-11T22:21:33.4500814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4500987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4501425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4501626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4501684Z 
2023-01-11T22:21:33.4501799Z Running tests...
2023-01-11T22:21:33.4502044Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4502358Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4502614Z   test_scatter_complex (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4502831Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 49849
2023-01-11T22:21:33.4503051Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 49850
2023-01-11T22:21:33.4503422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4503600Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4503984Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4504160Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4504525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4504697Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4505067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4505254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4505500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4505748Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4506147Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4506544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4506757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4506982Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4507316Z STAGE:2023-01-11 22:18:48 49850:49850 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4507641Z STAGE:2023-01-11 22:18:48 49849:49849 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4508187Z STAGE:2023-01-11 22:18:48 49850:49850 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:18:48 49849:49849 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4508208Z 
2023-01-11T22:21:33.4508776Z STAGE:2023-01-11 22:18:48 49850:49850 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:18:48 49849:49849 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4508799Z 
2023-01-11T22:21:33.4509117Z STAGE:2023-01-11 22:18:48 49850:49850 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4509447Z STAGE:2023-01-11 22:18:48 49849:49849 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4509777Z STAGE:2023-01-11 22:18:48 49850:49850 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4510098Z STAGE:2023-01-11 22:18:48 49849:49849 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4510442Z STAGE:2023-01-11 22:18:48 49850:49850 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4510866Z STAGE:2023-01-11 22:18:48 49849:49849 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4510974Z ok (4.228s)
2023-01-11T22:21:33.4510994Z 
2023-01-11T22:21:33.4511304Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4511416Z Ran 1 test in 4.228s
2023-01-11T22:21:33.4511435Z 
2023-01-11T22:21:33.4511528Z OK
2023-01-11T22:21:33.4511547Z 
2023-01-11T22:21:33.4511671Z Generating XML reports...
2023-01-11T22:21:33.4512130Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221845.xml
2023-01-11T22:21:33.4512500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4512659Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4513037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4513234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4513254Z 
2023-01-11T22:21:33.4513358Z Running tests...
2023-01-11T22:21:33.4513623Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4513937Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4514191Z   test_scatter_cuda (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2023-01-11T22:21:33.4514211Z 
2023-01-11T22:21:33.4514470Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4514578Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4514597Z 
2023-01-11T22:21:33.4514686Z OK (skipped=1)
2023-01-11T22:21:33.4514705Z 
2023-01-11T22:21:33.4514824Z Generating XML reports...
2023-01-11T22:21:33.4515268Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221851.xml
2023-01-11T22:21:33.4515642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4515813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4516199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4516390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4516410Z 
2023-01-11T22:21:33.4516518Z Running tests...
2023-01-11T22:21:33.4516762Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4517078Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4517346Z   test_scatter_cuda_complex (__main__.TestDistBackendWithSpawn) ... skip: Only Nccl supports CUDA gather (0.002s)
2023-01-11T22:21:33.4517366Z 
2023-01-11T22:21:33.4517624Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4517734Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4517753Z 
2023-01-11T22:21:33.4517856Z OK (skipped=1)
2023-01-11T22:21:33.4517875Z 
2023-01-11T22:21:33.4517999Z Generating XML reports...
2023-01-11T22:21:33.4518446Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221854.xml
2023-01-11T22:21:33.4518816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4518974Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4519353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4519541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4519561Z 
2023-01-11T22:21:33.4519664Z Running tests...
2023-01-11T22:21:33.4519976Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4520302Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4520562Z   test_scatter_full_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4520831Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50028
2023-01-11T22:21:33.4521054Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50029
2023-01-11T22:21:33.4521410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4521592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4521969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4522157Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4522523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4522693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4523069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4523259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4523489Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4523733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4524132Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4524755Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4524989Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4525214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4525449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.4525694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.4526092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4526485Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4526816Z STAGE:2023-01-11 22:19:00 50028:50028 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4527129Z STAGE:2023-01-11 22:19:00 50029:50029 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4527674Z STAGE:2023-01-11 22:19:00 50028:50028 ActivityProfilerController.cpp:306] Completed Stage: CollectionSTAGE:2023-01-11 22:19:00 50029:50029 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4527698Z 
2023-01-11T22:21:33.4528271Z STAGE:2023-01-11 22:19:00 50028:50028 ActivityProfilerController.cpp:310] Completed Stage: Post ProcessingSTAGE:2023-01-11 22:19:00 50029:50029 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4528292Z 
2023-01-11T22:21:33.4528614Z STAGE:2023-01-11 22:19:00 50028:50028 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4528929Z STAGE:2023-01-11 22:19:00 50029:50029 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4529258Z STAGE:2023-01-11 22:19:00 50029:50029 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4529663Z STAGE:2023-01-11 22:19:00 50028:50028 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4530021Z STAGE:2023-01-11 22:19:00 50029:50029 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4530430Z STAGE:2023-01-11 22:19:00 50028:50028 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4530529Z ok (4.236s)
2023-01-11T22:21:33.4530549Z 
2023-01-11T22:21:33.4530812Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4530905Z Ran 1 test in 4.236s
2023-01-11T22:21:33.4530925Z 
2023-01-11T22:21:33.4531016Z OK
2023-01-11T22:21:33.4531035Z 
2023-01-11T22:21:33.4531157Z Generating XML reports...
2023-01-11T22:21:33.4531609Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221856.xml
2023-01-11T22:21:33.4531976Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4532155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4532535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4532725Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4532745Z 
2023-01-11T22:21:33.4532850Z Running tests...
2023-01-11T22:21:33.4533095Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4533406Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4533656Z   test_scatter_group (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4533869Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50147
2023-01-11T22:21:33.4534081Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50148
2023-01-11T22:21:33.4534454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4534629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4535008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4535181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4535541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4535711Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4536079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4536258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4536502Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4536744Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4537142Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4537541Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4537755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4537980Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4538130Z skip: Skipped due to small world size. (4.222s)
2023-01-11T22:21:33.4538150Z 
2023-01-11T22:21:33.4538413Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4538520Z Ran 1 test in 4.223s
2023-01-11T22:21:33.4538540Z 
2023-01-11T22:21:33.4538641Z OK (skipped=1)
2023-01-11T22:21:33.4538711Z 
2023-01-11T22:21:33.4538838Z Generating XML reports...
2023-01-11T22:21:33.4539294Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221903.xml
2023-01-11T22:21:33.4539699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4539870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4540246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4540435Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4540455Z 
2023-01-11T22:21:33.4540561Z Running tests...
2023-01-11T22:21:33.4540813Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4541124Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4541389Z   test_scatter_object_list (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4541604Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50256
2023-01-11T22:21:33.4541809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50257
2023-01-11T22:21:33.4542181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4542351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4542730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4542920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4543283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4543452Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4543826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4544000Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4544244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4544482Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4544880Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4545275Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4545501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4545732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4545830Z ok (4.205s)
2023-01-11T22:21:33.4545850Z 
2023-01-11T22:21:33.4546115Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4546209Z Ran 1 test in 4.205s
2023-01-11T22:21:33.4546228Z 
2023-01-11T22:21:33.4546312Z OK
2023-01-11T22:21:33.4546332Z 
2023-01-11T22:21:33.4546452Z Generating XML reports...
2023-01-11T22:21:33.4546895Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221910.xml
2023-01-11T22:21:33.4547263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4547433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4547809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4548043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4548065Z 
2023-01-11T22:21:33.4548175Z Running tests...
2023-01-11T22:21:33.4548422Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4548780Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4549025Z   test_send_recv (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4549244Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50365
2023-01-11T22:21:33.4549459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50366
2023-01-11T22:21:33.4549830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4550003Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4550380Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4550553Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4550916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4551088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4551458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4551641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4551885Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4552122Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4552524Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4552919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4553136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4553362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4553460Z ok (4.100s)
2023-01-11T22:21:33.4553479Z 
2023-01-11T22:21:33.4553738Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4553846Z Ran 1 test in 4.100s
2023-01-11T22:21:33.4553865Z 
2023-01-11T22:21:33.4553954Z OK
2023-01-11T22:21:33.4553974Z 
2023-01-11T22:21:33.4554095Z Generating XML reports...
2023-01-11T22:21:33.4554545Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221916.xml
2023-01-11T22:21:33.4554901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4555073Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4555455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4555646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4555665Z 
2023-01-11T22:21:33.4555769Z Running tests...
2023-01-11T22:21:33.4556028Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4556336Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4556597Z   test_send_recv_any_source (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4556811Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50474
2023-01-11T22:21:33.4557060Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50475
2023-01-11T22:21:33.4557442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4557660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4558040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4558220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4558579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4558747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4559117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4559291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4559533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4559775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4560179Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4560575Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4560803Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4561029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4561128Z ok (4.242s)
2023-01-11T22:21:33.4561148Z 
2023-01-11T22:21:33.4561409Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4561505Z Ran 1 test in 4.242s
2023-01-11T22:21:33.4561525Z 
2023-01-11T22:21:33.4561616Z OK
2023-01-11T22:21:33.4561635Z 
2023-01-11T22:21:33.4561756Z Generating XML reports...
2023-01-11T22:21:33.4562212Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221923.xml
2023-01-11T22:21:33.4562581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4562753Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4563131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4563319Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4563338Z 
2023-01-11T22:21:33.4563442Z Running tests...
2023-01-11T22:21:33.4563686Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4563998Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4564504Z   test_send_recv_any_source_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4564734Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50583
2023-01-11T22:21:33.4564952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50584
2023-01-11T22:21:33.4565331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4565501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4565879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4566052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4566501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4566683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4567063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4567300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4567539Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4567782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4568184Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4568580Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4568798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4569021Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4569360Z STAGE:2023-01-11 22:19:34 50583:50583 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4569689Z STAGE:2023-01-11 22:19:34 50584:50584 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4570024Z STAGE:2023-01-11 22:19:34 50583:50583 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4570368Z STAGE:2023-01-11 22:19:34 50583:50583 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4570698Z STAGE:2023-01-11 22:19:34 50584:50584 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4571044Z STAGE:2023-01-11 22:19:34 50584:50584 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4571128Z ok (4.224s)
2023-01-11T22:21:33.4571166Z 
2023-01-11T22:21:33.4571414Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4571522Z Ran 1 test in 4.224s
2023-01-11T22:21:33.4571544Z 
2023-01-11T22:21:33.4571630Z OK
2023-01-11T22:21:33.4571649Z 
2023-01-11T22:21:33.4571771Z Generating XML reports...
2023-01-11T22:21:33.4572224Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221930.xml
2023-01-11T22:21:33.4572592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4572764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4573138Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4573314Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4573333Z 
2023-01-11T22:21:33.4573442Z Running tests...
2023-01-11T22:21:33.4573700Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4574009Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4574293Z   test_send_recv_any_source_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4574509Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50696
2023-01-11T22:21:33.4574724Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50697
2023-01-11T22:21:33.4575093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4575249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4575625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4575859Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4576231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4576447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4576817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4577002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4577244Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4577486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4577868Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4578261Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4578489Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4578715Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4579048Z STAGE:2023-01-11 22:19:40 50696:50696 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4579367Z STAGE:2023-01-11 22:19:40 50697:50697 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4579695Z STAGE:2023-01-11 22:19:40 50696:50696 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4580041Z STAGE:2023-01-11 22:19:40 50696:50696 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4580367Z STAGE:2023-01-11 22:19:40 50697:50697 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4580698Z STAGE:2023-01-11 22:19:40 50697:50697 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4580797Z ok (4.220s)
2023-01-11T22:21:33.4580817Z 
2023-01-11T22:21:33.4581079Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4581186Z Ran 1 test in 4.220s
2023-01-11T22:21:33.4581206Z 
2023-01-11T22:21:33.4581294Z OK
2023-01-11T22:21:33.4581313Z 
2023-01-11T22:21:33.4581434Z Generating XML reports...
2023-01-11T22:21:33.4581883Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221937.xml
2023-01-11T22:21:33.4582251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4582419Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4582782Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4582976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4582995Z 
2023-01-11T22:21:33.4583099Z Running tests...
2023-01-11T22:21:33.4583354Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4583666Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4583940Z   test_send_recv_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4584156Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 50809
2023-01-11T22:21:33.4584372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 50810
2023-01-11T22:21:33.4584723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4584895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4585319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4585511Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4585927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4586096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4586473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4586657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4586886Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4587130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4587530Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4587922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4588152Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4588376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4588705Z STAGE:2023-01-11 22:19:47 50809:50809 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4589023Z STAGE:2023-01-11 22:19:47 50810:50810 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4589354Z STAGE:2023-01-11 22:19:47 50810:50810 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4589685Z STAGE:2023-01-11 22:19:47 50810:50810 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4590022Z STAGE:2023-01-11 22:19:47 50809:50809 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4590366Z STAGE:2023-01-11 22:19:47 50809:50809 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4590468Z ok (4.221s)
2023-01-11T22:21:33.4590488Z 
2023-01-11T22:21:33.4590748Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4590858Z Ran 1 test in 4.222s
2023-01-11T22:21:33.4590878Z 
2023-01-11T22:21:33.4590966Z OK
2023-01-11T22:21:33.4590985Z 
2023-01-11T22:21:33.4591105Z Generating XML reports...
2023-01-11T22:21:33.4591558Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221943.xml
2023-01-11T22:21:33.4591912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4592082Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4592461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4592648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4592671Z 
2023-01-11T22:21:33.4592775Z Running tests...
2023-01-11T22:21:33.4593033Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4593342Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4593575Z   test_send_recv_nccl (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2023-01-11T22:21:33.4593594Z 
2023-01-11T22:21:33.4593854Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4593946Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4593965Z 
2023-01-11T22:21:33.4594069Z OK (skipped=1)
2023-01-11T22:21:33.4594088Z 
2023-01-11T22:21:33.4594208Z Generating XML reports...
2023-01-11T22:21:33.4594708Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221950.xml
2023-01-11T22:21:33.4595092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4595311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4595694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4595884Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4595904Z 
2023-01-11T22:21:33.4595993Z Running tests...
2023-01-11T22:21:33.4596255Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4596567Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4596834Z   test_send_recv_nccl_autograd_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2023-01-11T22:21:33.4596854Z 
2023-01-11T22:21:33.4597111Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4597223Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4597243Z 
2023-01-11T22:21:33.4597350Z OK (skipped=1)
2023-01-11T22:21:33.4597368Z 
2023-01-11T22:21:33.4597488Z Generating XML reports...
2023-01-11T22:21:33.4597931Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221952.xml
2023-01-11T22:21:33.4598281Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4598455Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4598830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4599019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4599042Z 
2023-01-11T22:21:33.4599148Z Running tests...
2023-01-11T22:21:33.4599406Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4599722Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4599980Z   test_send_recv_nccl_torch_profiler (__main__.TestDistBackendWithSpawn) ... skip: NCCL Send Recv Only (0.002s)
2023-01-11T22:21:33.4599999Z 
2023-01-11T22:21:33.4600255Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4600347Z Ran 1 test in 0.002s
2023-01-11T22:21:33.4600367Z 
2023-01-11T22:21:33.4600469Z OK (skipped=1)
2023-01-11T22:21:33.4600489Z 
2023-01-11T22:21:33.4600603Z Generating XML reports...
2023-01-11T22:21:33.4601051Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221955.xml
2023-01-11T22:21:33.4601421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4601598Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4601973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4602163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4602183Z 
2023-01-11T22:21:33.4602286Z Running tests...
2023-01-11T22:21:33.4602533Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4602843Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4603112Z   test_send_recv_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4603328Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51021
2023-01-11T22:21:33.4603589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51022
2023-01-11T22:21:33.4603963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4604179Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4604787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4604961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4605326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4605498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4605867Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4606054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4606305Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4606549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4606953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4607347Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4607563Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4607783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4608119Z STAGE:2023-01-11 22:20:01 51021:51021 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4608438Z STAGE:2023-01-11 22:20:01 51022:51022 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4608772Z STAGE:2023-01-11 22:20:01 51022:51022 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4609115Z STAGE:2023-01-11 22:20:01 51022:51022 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4609445Z STAGE:2023-01-11 22:20:01 51021:51021 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4609789Z STAGE:2023-01-11 22:20:01 51021:51021 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4609872Z ok (4.206s)
2023-01-11T22:21:33.4609905Z 
2023-01-11T22:21:33.4610152Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4610259Z Ran 1 test in 4.206s
2023-01-11T22:21:33.4610278Z 
2023-01-11T22:21:33.4610368Z OK
2023-01-11T22:21:33.4610387Z 
2023-01-11T22:21:33.4610546Z Generating XML reports...
2023-01-11T22:21:33.4611005Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221957.xml
2023-01-11T22:21:33.4611374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4611552Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4611929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4612103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4612137Z 
2023-01-11T22:21:33.4612226Z Running tests...
2023-01-11T22:21:33.4612485Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4612799Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4613057Z   test_send_recv_with_tag (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4613345Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51134
2023-01-11T22:21:33.4613572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51135
2023-01-11T22:21:33.4613945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4614158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4614538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4614726Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4615086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4615259Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4615634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4615826Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4616069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4616314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4616698Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4617092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4617319Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4617548Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4617646Z ok (4.341s)
2023-01-11T22:21:33.4617666Z 
2023-01-11T22:21:33.4617931Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4618036Z Ran 1 test in 4.342s
2023-01-11T22:21:33.4618056Z 
2023-01-11T22:21:33.4618143Z OK
2023-01-11T22:21:33.4618165Z 
2023-01-11T22:21:33.4618283Z Generating XML reports...
2023-01-11T22:21:33.4618720Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222004.xml
2023-01-11T22:21:33.4619083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4619255Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4619630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4619816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4619836Z 
2023-01-11T22:21:33.4619940Z Running tests...
2023-01-11T22:21:33.4620206Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4620519Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4620790Z   test_send_recv_with_tag_autograd_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4621006Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51243
2023-01-11T22:21:33.4621218Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51244
2023-01-11T22:21:33.4621586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4621759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4622136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4622322Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4622772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4622953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4623371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4623554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4623799Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4624197Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4624434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4624832Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4625060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4625285Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4625620Z STAGE:2023-01-11 22:20:15 51243:51243 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4625928Z STAGE:2023-01-11 22:20:15 51244:51244 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4626261Z STAGE:2023-01-11 22:20:15 51243:51243 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4626603Z STAGE:2023-01-11 22:20:15 51243:51243 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4626935Z STAGE:2023-01-11 22:20:15 51244:51244 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4627282Z STAGE:2023-01-11 22:20:15 51244:51244 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4627381Z ok (4.230s)
2023-01-11T22:21:33.4627401Z 
2023-01-11T22:21:33.4627657Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4627769Z Ran 1 test in 4.230s
2023-01-11T22:21:33.4627788Z 
2023-01-11T22:21:33.4627863Z OK
2023-01-11T22:21:33.4627898Z 
2023-01-11T22:21:33.4628003Z Generating XML reports...
2023-01-11T22:21:33.4628455Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222011.xml
2023-01-11T22:21:33.4628823Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4628992Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4629367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4629557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4629577Z 
2023-01-11T22:21:33.4629678Z Running tests...
2023-01-11T22:21:33.4629932Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4630235Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4630510Z   test_send_recv_with_tag_torch_profiler (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4630721Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51356
2023-01-11T22:21:33.4630935Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51357
2023-01-11T22:21:33.4631301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4631468Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4631905Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4632101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4632456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4632663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4633036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4633220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4633462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4633707Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4634108Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4634499Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4634723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4634937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4635271Z STAGE:2023-01-11 22:20:21 51357:51357 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4635589Z STAGE:2023-01-11 22:20:21 51356:51356 ActivityProfilerController.cpp:300] Completed Stage: Warm Up
2023-01-11T22:21:33.4635918Z STAGE:2023-01-11 22:20:21 51356:51356 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4636263Z STAGE:2023-01-11 22:20:21 51356:51356 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4636596Z STAGE:2023-01-11 22:20:21 51357:51357 ActivityProfilerController.cpp:306] Completed Stage: Collection
2023-01-11T22:21:33.4636933Z STAGE:2023-01-11 22:20:21 51357:51357 ActivityProfilerController.cpp:310] Completed Stage: Post Processing
2023-01-11T22:21:33.4637033Z ok (4.223s)
2023-01-11T22:21:33.4637052Z 
2023-01-11T22:21:33.4637312Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4637405Z Ran 1 test in 4.224s
2023-01-11T22:21:33.4637425Z 
2023-01-11T22:21:33.4637515Z OK
2023-01-11T22:21:33.4637534Z 
2023-01-11T22:21:33.4637653Z Generating XML reports...
2023-01-11T22:21:33.4638096Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222018.xml
2023-01-11T22:21:33.4638461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4638635Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4639019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4639209Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4639232Z 
2023-01-11T22:21:33.4639330Z Running tests...
2023-01-11T22:21:33.4639576Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4639884Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4640149Z   test_sparse_all_reduce_sum (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4640365Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51469
2023-01-11T22:21:33.4640578Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51470
2023-01-11T22:21:33.4640947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4641171Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4641557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4641803Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4642167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4642338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4642707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4642894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4643136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4643375Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4643776Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4644178Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4644710Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4644940Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4645033Z ok (4.241s)
2023-01-11T22:21:33.4645053Z 
2023-01-11T22:21:33.4645325Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4645435Z Ran 1 test in 4.241s
2023-01-11T22:21:33.4645455Z 
2023-01-11T22:21:33.4645540Z OK
2023-01-11T22:21:33.4645559Z 
2023-01-11T22:21:33.4645676Z Generating XML reports...
2023-01-11T22:21:33.4646132Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222024.xml
2023-01-11T22:21:33.4646488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4646663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4647039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4647229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4647249Z 
2023-01-11T22:21:33.4647354Z Running tests...
2023-01-11T22:21:33.4647608Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4647915Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4648179Z   test_sparse_all_reduce_sum_cuda (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4648395Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51668
2023-01-11T22:21:33.4648597Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51669
2023-01-11T22:21:33.4648968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4649140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4649513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4649704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4650065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4650235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4650685Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4650868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4651113Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4651408Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4651808Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4652197Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4652424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4652650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4652747Z ok (5.050s)
2023-01-11T22:21:33.4652767Z 
2023-01-11T22:21:33.4653027Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4653122Z Ran 1 test in 5.050s
2023-01-11T22:21:33.4653142Z 
2023-01-11T22:21:33.4653229Z OK
2023-01-11T22:21:33.4653252Z 
2023-01-11T22:21:33.4653366Z Generating XML reports...
2023-01-11T22:21:33.4653816Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222031.xml
2023-01-11T22:21:33.4654185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4654355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4654732Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4654918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4654937Z 
2023-01-11T22:21:33.4655026Z Running tests...
2023-01-11T22:21:33.4655292Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4655605Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4655874Z   test_stateless_api_with_ddp (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4656091Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51869
2023-01-11T22:21:33.4656302Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51870
2023-01-11T22:21:33.4656672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4656843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4657214Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4657387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4657745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4657916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4658288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4658470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4658715Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4658956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4659357Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4659795Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4660014Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4660242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4660383Z ok (5.520s)
2023-01-11T22:21:33.4660403Z 
2023-01-11T22:21:33.4660666Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4660775Z Ran 1 test in 5.520s
2023-01-11T22:21:33.4660794Z 
2023-01-11T22:21:33.4660882Z OK
2023-01-11T22:21:33.4660901Z 
2023-01-11T22:21:33.4661022Z Generating XML reports...
2023-01-11T22:21:33.4661474Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222039.xml
2023-01-11T22:21:33.4661832Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4662006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4662384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4662575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4662594Z 
2023-01-11T22:21:33.4662698Z Running tests...
2023-01-11T22:21:33.4662959Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4663268Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4663524Z   test_static_graph_api_cpu (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4663727Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 51984
2023-01-11T22:21:33.4663942Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 51985
2023-01-11T22:21:33.4664312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4664481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4664858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4665048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4665409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4665576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4665945Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4666113Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4666355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4666600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4667001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4667401Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4667626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4667846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4667941Z ok (4.138s)
2023-01-11T22:21:33.4667960Z 
2023-01-11T22:21:33.4668219Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4668313Z Ran 1 test in 4.138s
2023-01-11T22:21:33.4668333Z 
2023-01-11T22:21:33.4668420Z OK
2023-01-11T22:21:33.4668439Z 
2023-01-11T22:21:33.4668561Z Generating XML reports...
2023-01-11T22:21:33.4669059Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222047.xml
2023-01-11T22:21:33.4669441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4669657Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4670033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4670220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4670240Z 
2023-01-11T22:21:33.4670331Z Running tests...
2023-01-11T22:21:33.4670592Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4670903Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4671150Z   test_sync_bn_logged (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4671367Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52097
2023-01-11T22:21:33.4671577Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52098
2023-01-11T22:21:33.4671951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4672118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4672492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4672667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4673035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4673206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4673585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4673772Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4674014Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4674256Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4674647Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4675028Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4675254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4675477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4675577Z ok (5.107s)
2023-01-11T22:21:33.4675596Z 
2023-01-11T22:21:33.4675858Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4675962Z Ran 1 test in 5.107s
2023-01-11T22:21:33.4675984Z 
2023-01-11T22:21:33.4676073Z OK
2023-01-11T22:21:33.4676092Z 
2023-01-11T22:21:33.4676212Z Generating XML reports...
2023-01-11T22:21:33.4676655Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222053.xml
2023-01-11T22:21:33.4677009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4677182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4677556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4677744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4677764Z 
2023-01-11T22:21:33.4677913Z Running tests...
2023-01-11T22:21:33.4678182Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4678496Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4678833Z   test_undefined_grad_parity_unused_parameters (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4679036Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52208
2023-01-11T22:21:33.4679250Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52209
2023-01-11T22:21:33.4679622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4679792Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4680169Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4680358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4680721Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4680896Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4681262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4681430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4681669Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4681907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4682302Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4682695Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4682923Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4683148Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4683932Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:21:33.4684891Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:21:33.4684996Z ok (5.634s)
2023-01-11T22:21:33.4685016Z 
2023-01-11T22:21:33.4685283Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4685395Z Ran 1 test in 5.634s
2023-01-11T22:21:33.4685414Z 
2023-01-11T22:21:33.4685489Z OK
2023-01-11T22:21:33.4685523Z 
2023-01-11T22:21:33.4685629Z Generating XML reports...
2023-01-11T22:21:33.4686085Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222101.xml
2023-01-11T22:21:33.4686534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4686717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4687157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4687347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4687367Z 
2023-01-11T22:21:33.4687467Z Running tests...
2023-01-11T22:21:33.4687727Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4688026Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4688302Z   test_verify_model_across_rank_with_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4688520Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52323
2023-01-11T22:21:33.4688737Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52324
2023-01-11T22:21:33.4689107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4689277Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4689647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4689834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4690180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4690350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4690722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4690911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4691152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4691390Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4691791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4692182Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4692413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4692624Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4692860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.4693102Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.4693494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4693891Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4694132Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:21:33.4694363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:21:33.4694746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:21:33.4695135Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:21:33.4695219Z ok (10.117s)
2023-01-11T22:21:33.4695252Z 
2023-01-11T22:21:33.4695561Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4695677Z Ran 1 test in 10.118s
2023-01-11T22:21:33.4695734Z 
2023-01-11T22:21:33.4695818Z OK
2023-01-11T22:21:33.4695837Z 
2023-01-11T22:21:33.4695959Z Generating XML reports...
2023-01-11T22:21:33.4696414Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222109.xml
2023-01-11T22:21:33.4696784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4696957Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4697333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4697509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4697528Z 
2023-01-11T22:21:33.4697631Z Running tests...
2023-01-11T22:21:33.4697887Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4698201Z Test results will be stored in test-reports/dist-gloo/distributed.test_distributed_spawn
2023-01-11T22:21:33.4698492Z   test_verify_model_across_rank_without_logger (__main__.TestDistBackendWithSpawn) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:21:33.4698706Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 52446
2023-01-11T22:21:33.4698921Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 52447
2023-01-11T22:21:33.4699285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4699441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4699815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4700005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4700361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:21:33.4700534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:21:33.4700911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:21:33.4701096Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:21:33.4701342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:21:33.4701577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:21:33.4701961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4702350Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:21:33.4702579Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:21:33.4702807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:21:33.4703047Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:21:33.4703287Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:21:33.4703682Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4704066Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:21:33.4704302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:21:33.4704574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:21:33.4704972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:21:33.4705411Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:21:33.4705502Z ok (10.151s)
2023-01-11T22:21:33.4705521Z 
2023-01-11T22:21:33.4705782Z ----------------------------------------------------------------------
2023-01-11T22:21:33.4705892Z Ran 1 test in 10.151s
2023-01-11T22:21:33.4705912Z 
2023-01-11T22:21:33.4705999Z OK
2023-01-11T22:21:33.4706019Z 
2023-01-11T22:21:33.4706140Z Generating XML reports...
2023-01-11T22:21:33.4706587Z Generated XML report: test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222122.xml
2023-01-11T22:21:33.4706606Z 
2023-01-11T22:21:33.4707083Z ##[endgroup]
2023-01-11T22:21:33.4707560Z FINISHED PRINTING LOG FILE of distributed/test_distributed_spawn (/var/lib/jenkins/workspace/test/test-reports/distributed-test_distributed_spawn_fvmfkweh)
2023-01-11T22:21:33.4707584Z 
2023-01-11T22:21:33.4707707Z Shard 1: ucc should be run in 3
2023-01-11T22:21:33.4707980Z Running distributed/rpc/test_tensorpipe_agent ... [2023-01-11 22:21:33.214597]
2023-01-11T22:21:33.4708492Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:21:33.214841]
2023-01-11T22:21:35.8085494Z 
2023-01-11T22:21:35.8086206Z Expand the folded group to see the log file of distributed/rpc/test_tensorpipe_agent
2023-01-11T22:21:35.8087330Z ##[group]PRINTING LOG FILE of distributed/rpc/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_tensorpipe_agent_3c9gyp_6)
2023-01-11T22:21:35.8087953Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbi0h1tsp
2023-01-11T22:21:35.8088510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbi0h1tsp/_remote_module_non_scriptable.py
2023-01-11T22:21:35.8088848Z 
2023-01-11T22:21:35.8089153Z ##[endgroup]
2023-01-11T22:21:35.8089879Z FINISHED PRINTING LOG FILE of distributed/rpc/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-test_tensorpipe_agent_3c9gyp_6)
2023-01-11T22:21:35.8090258Z 
2023-01-11T22:21:35.8090565Z Running distributed/pipeline/sync/test_transparency ... [2023-01-11 22:21:35.808649]
2023-01-11T22:21:35.8091663Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_transparency.py', '-v'] ... [2023-01-11 22:21:35.808905]
2023-01-11T22:21:38.7725709Z 
2023-01-11T22:21:38.7726395Z Expand the folded group to see the log file of distributed/pipeline/sync/test_transparency
2023-01-11T22:21:38.7727422Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_transparency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_transparency_63dkca1g)
2023-01-11T22:21:38.7727985Z ============================= test session starts ==============================
2023-01-11T22:21:38.7728821Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:21:38.7729183Z cachedir: .pytest_cache
2023-01-11T22:21:38.7730022Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:21:38.7730483Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:21:38.7731010Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:21:38.7731587Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:21:38.7732200Z collecting ... collected 1 item
2023-01-11T22:21:38.7732693Z Running 1 items in this shard: test/distributed/pipeline/sync/test_transparency.py::test_simple_linears
2023-01-11T22:21:38.7732978Z 
2023-01-11T22:21:38.7733454Z distributed/pipeline/sync/test_transparency.py::test_simple_linears PASSED [100%]
2023-01-11T22:21:38.7733733Z 
2023-01-11T22:21:38.7733876Z ============================== 1 passed in 0.15s ===============================
2023-01-11T22:21:38.7734162Z 
2023-01-11T22:21:38.7734480Z ##[endgroup]
2023-01-11T22:21:38.7735165Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_transparency (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_transparency_63dkca1g)
2023-01-11T22:21:38.7735572Z 
2023-01-11T22:21:38.7735851Z Running distributed/pipeline/sync/test_pipe ... [2023-01-11 22:21:38.772650]
2023-01-11T22:21:38.7736438Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_pipe.py', '-v'] ... [2023-01-11 22:21:38.772932]
2023-01-11T22:21:46.2127450Z 
2023-01-11T22:21:46.2128128Z Expand the folded group to see the log file of distributed/pipeline/sync/test_pipe
2023-01-11T22:21:46.2129188Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_pipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipe_9acghfbm)
2023-01-11T22:21:46.2129732Z ============================= test session starts ==============================
2023-01-11T22:21:46.2130371Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:21:46.2130708Z cachedir: .pytest_cache
2023-01-11T22:21:46.2131385Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:21:46.2131975Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:21:46.2132291Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:21:46.2133159Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:21:46.2133574Z collecting ... collected 56 items
2023-01-11T22:21:46.2140094Z Running 56 items in this shard: test/distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc, test/distributed/pipeline/sync/test_pipe.py::test_parameters, test/distributed/pipeline/sync/test_pipe.py::test_public_attrs, test/distributed/pipeline/sync/test_pipe.py::test_sequential_like, test/distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible, test/distributed/pipeline/sync/test_pipe.py::test_batch_size_small, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval, test/distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input, test/distributed/pipeline/sync/test_pipe.py::test_no_grad, test/distributed/pipeline/sync/test_pipe.py::test_exception, test/distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap, test/distributed/pipeline/sync/test_pipe.py::test_nested_input, test/distributed/pipeline/sync/test_pipe.py::test_input_pair, test/distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input, test/distributed/pipeline/sync/test_pipe.py::test_input_singleton, test/distributed/pipeline/sync/test_pipe.py::test_input_varargs, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor, test/distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence, test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always], test/distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always], test/distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always], test/distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[never], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[always], test/distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never], test/distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always], test/distributed/pipeline/sync/test_pipe.py::test_devices, test/distributed/pipeline/sync/test_pipe.py::test_partitions, test/distributed/pipeline/sync/test_pipe.py::test_merged_partitions, test/distributed/pipeline/sync/test_pipe.py::test_deny_moving, test/distributed/pipeline/sync/test_pipe.py::test_empty_module, test/distributed/pipeline/sync/test_pipe.py::test_named_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules, test/distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device, test/distributed/pipeline/sync/test_pipe.py::test_forward_lockstep, test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always], test/distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last], test/distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device, test/distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper
2023-01-11T22:21:46.2145803Z 
2023-01-11T22:21:46.2146039Z distributed/pipeline/sync/test_pipe.py::test_pipe_without_rpc PASSED     [  1%]
2023-01-11T22:21:46.2146489Z distributed/pipeline/sync/test_pipe.py::test_parameters PASSED           [  3%]
2023-01-11T22:21:46.2146908Z distributed/pipeline/sync/test_pipe.py::test_public_attrs PASSED         [  5%]
2023-01-11T22:21:46.2147333Z distributed/pipeline/sync/test_pipe.py::test_sequential_like PASSED      [  7%]
2023-01-11T22:21:46.2147763Z distributed/pipeline/sync/test_pipe.py::test_chunks_less_than_1 PASSED   [  8%]
2023-01-11T22:21:46.2148191Z distributed/pipeline/sync/test_pipe.py::test_batch_size_indivisible PASSED [ 10%]
2023-01-11T22:21:46.2148635Z distributed/pipeline/sync/test_pipe.py::test_batch_size_small PASSED     [ 12%]
2023-01-11T22:21:46.2149065Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode PASSED      [ 14%]
2023-01-11T22:21:46.2149513Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_invalid PASSED [ 16%]
2023-01-11T22:21:46.2149969Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_mode_when_chunks_1 PASSED [ 17%]
2023-01-11T22:21:46.2150433Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_eval PASSED      [ 19%]
2023-01-11T22:21:46.2150885Z distributed/pipeline/sync/test_pipe.py::test_checkpoint_non_float_input PASSED [ 21%]
2023-01-11T22:21:46.2151310Z distributed/pipeline/sync/test_pipe.py::test_no_grad PASSED              [ 23%]
2023-01-11T22:21:46.2151724Z distributed/pipeline/sync/test_pipe.py::test_exception PASSED            [ 25%]
2023-01-11T22:21:46.2152166Z distributed/pipeline/sync/test_pipe.py::test_exception_early_stop_asap PASSED [ 26%]
2023-01-11T22:21:46.2152612Z distributed/pipeline/sync/test_pipe.py::test_nested_input PASSED         [ 28%]
2023-01-11T22:21:46.2153018Z distributed/pipeline/sync/test_pipe.py::test_input_pair PASSED           [ 30%]
2023-01-11T22:21:46.2153442Z distributed/pipeline/sync/test_pipe.py::test_multi_sequence_input PASSED [ 32%]
2023-01-11T22:21:46.2153873Z distributed/pipeline/sync/test_pipe.py::test_input_singleton PASSED      [ 33%]
2023-01-11T22:21:46.2154343Z distributed/pipeline/sync/test_pipe.py::test_input_varargs PASSED        [ 35%]
2023-01-11T22:21:46.2154771Z distributed/pipeline/sync/test_pipe.py::test_non_tensor PASSED           [ 37%]
2023-01-11T22:21:46.2155237Z distributed/pipeline/sync/test_pipe.py::test_non_tensor_sequence PASSED  [ 39%]
2023-01-11T22:21:46.2155683Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[never] PASSED [ 41%]
2023-01-11T22:21:46.2156115Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[always] PASSED [ 42%]
2023-01-11T22:21:46.2156577Z distributed/pipeline/sync/test_pipe.py::test_valid_non_tensor[except_last] PASSED [ 44%]
2023-01-11T22:21:46.2157035Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[never] PASSED [ 46%]
2023-01-11T22:21:46.2157467Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[always] PASSED [ 48%]
2023-01-11T22:21:46.2157928Z distributed/pipeline/sync/test_pipe.py::test_no_tensor_output[except_last] PASSED [ 50%]
2023-01-11T22:21:46.2158391Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[never] PASSED [ 51%]
2023-01-11T22:21:46.2158844Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[always] PASSED [ 53%]
2023-01-11T22:21:46.2159292Z distributed/pipeline/sync/test_pipe.py::test_uneven_batch_size[except_last] PASSED [ 55%]
2023-01-11T22:21:46.2159744Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[never] PASSED      [ 57%]
2023-01-11T22:21:46.2160171Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[always] PASSED     [ 58%]
2023-01-11T22:21:46.2160589Z distributed/pipeline/sync/test_pipe.py::test_no_chunk[except_last] PASSED [ 60%]
2023-01-11T22:21:46.2161038Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[never] PASSED [ 62%]
2023-01-11T22:21:46.2161500Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[always] PASSED [ 64%]
2023-01-11T22:21:46.2161968Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm[except_last] PASSED [ 66%]
2023-01-11T22:21:46.2162435Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[never] PASSED [ 67%]
2023-01-11T22:21:46.2162916Z distributed/pipeline/sync/test_pipe.py::test_deferred_batch_norm_params[always] PASSED [ 69%]
2023-01-11T22:21:46.2163372Z distributed/pipeline/sync/test_pipe.py::test_devices PASSED              [ 71%]
2023-01-11T22:21:46.2163776Z distributed/pipeline/sync/test_pipe.py::test_partitions PASSED           [ 73%]
2023-01-11T22:21:46.2164484Z distributed/pipeline/sync/test_pipe.py::test_merged_partitions PASSED    [ 75%]
2023-01-11T22:21:46.2164929Z distributed/pipeline/sync/test_pipe.py::test_deny_moving PASSED          [ 76%]
2023-01-11T22:21:46.2165353Z distributed/pipeline/sync/test_pipe.py::test_empty_module PASSED         [ 78%]
2023-01-11T22:21:46.2165760Z distributed/pipeline/sync/test_pipe.py::test_named_children PASSED       [ 80%]
2023-01-11T22:21:46.2166209Z distributed/pipeline/sync/test_pipe.py::test_verify_module_non_sequential PASSED [ 82%]
2023-01-11T22:21:46.2166687Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_children PASSED [ 83%]
2023-01-11T22:21:46.2167152Z distributed/pipeline/sync/test_pipe.py::test_verify_module_params_on_same_device PASSED [ 85%]
2023-01-11T22:21:46.2167619Z distributed/pipeline/sync/test_pipe.py::test_verify_nested_modules PASSED [ 87%]
2023-01-11T22:21:46.2168113Z distributed/pipeline/sync/test_pipe.py::test_verify_module_duplicate_parameters_on_same_device PASSED [ 89%]
2023-01-11T22:21:46.2168594Z distributed/pipeline/sync/test_pipe.py::test_forward_lockstep PASSED     [ 91%]
2023-01-11T22:21:46.2169021Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[never] PASSED [ 92%]
2023-01-11T22:21:46.2169474Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[always] PASSED [ 94%]
2023-01-11T22:21:46.2169938Z distributed/pipeline/sync/test_pipe.py::test_multiple_inputs[except_last] PASSED [ 96%]
2023-01-11T22:21:46.2170376Z distributed/pipeline/sync/test_pipe.py::test_inputs_wrong_device PASSED  [ 98%]
2023-01-11T22:21:46.2170899Z distributed/pipeline/sync/test_pipe.py::test_with_device_wrapper PASSED  [100%]
2023-01-11T22:21:46.2171155Z 
2023-01-11T22:21:46.2171314Z ============================== 56 passed in 4.92s ==============================
2023-01-11T22:21:46.2171574Z 
2023-01-11T22:21:46.2171895Z ##[endgroup]
2023-01-11T22:21:46.2172537Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_pipe (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_pipe_9acghfbm)
2023-01-11T22:21:46.2172920Z 
2023-01-11T22:21:46.2173203Z Running distributed/pipeline/sync/test_inplace ... [2023-01-11 22:21:46.212935]
2023-01-11T22:21:46.2173826Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_inplace.py', '-v'] ... [2023-01-11 22:21:46.213222]
2023-01-11T22:21:48.7440639Z 
2023-01-11T22:21:48.7441514Z Expand the folded group to see the log file of distributed/pipeline/sync/test_inplace
2023-01-11T22:21:48.7442555Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_inplace (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_inplace_kmxnkaox)
2023-01-11T22:21:48.7443084Z ============================= test session starts ==============================
2023-01-11T22:21:48.7443711Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:21:48.7444071Z cachedir: .pytest_cache
2023-01-11T22:21:48.7445310Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:21:48.7445769Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:21:48.7446102Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:21:48.7446688Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:21:48.7447076Z collecting ... collected 3 items
2023-01-11T22:21:48.7447730Z Running 3 items in this shard: test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad, test/distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad
2023-01-11T22:21:48.7448233Z 
2023-01-11T22:21:48.7448468Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_requires_grad PASSED [ 33%]
2023-01-11T22:21:48.7448946Z distributed/pipeline/sync/test_inplace.py::test_inplace_on_not_requires_grad XFAIL [ 66%]
2023-01-11T22:21:48.7449392Z distributed/pipeline/sync/test_inplace.py::test_inplace_incorrect_grad XFAIL [100%]
2023-01-11T22:21:48.7449650Z 
2023-01-11T22:21:48.7449818Z ========================= 1 passed, 2 xfailed in 0.17s =========================
2023-01-11T22:21:48.7450025Z 
2023-01-11T22:21:48.7450348Z ##[endgroup]
2023-01-11T22:21:48.7450987Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_inplace (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_inplace_kmxnkaox)
2023-01-11T22:21:48.7451372Z 
2023-01-11T22:21:48.7451656Z Running distributed/pipeline/sync/test_copy ... [2023-01-11 22:21:48.744124]
2023-01-11T22:21:48.7452260Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_copy.py', '-v'] ... [2023-01-11 22:21:48.744408]
2023-01-11T22:21:53.2375036Z 
2023-01-11T22:21:53.2375780Z Expand the folded group to see the log file of distributed/pipeline/sync/test_copy
2023-01-11T22:21:53.2376739Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_copy (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_copy_6g3dkoff)
2023-01-11T22:21:53.2377300Z ============================= test session starts ==============================
2023-01-11T22:21:53.2378264Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:21:53.2378631Z cachedir: .pytest_cache
2023-01-11T22:21:53.2379202Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:21:53.2379947Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:21:53.2380574Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:21:53.2381176Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:21:53.2381687Z collecting ... collected 5 items
2023-01-11T22:21:53.2382470Z Running 5 items in this shard: test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu, test/distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda, test/distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors
2023-01-11T22:21:53.2383097Z 
2023-01-11T22:21:53.2383317Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cpu PASSED    [ 20%]
2023-01-11T22:21:53.2383764Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cpu_cuda PASSED   [ 40%]
2023-01-11T22:21:53.2384191Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cpu PASSED   [ 60%]
2023-01-11T22:21:53.2384623Z distributed/pipeline/sync/test_copy.py::test_copy_wait_cuda_cuda PASSED  [ 80%]
2023-01-11T22:21:53.2385076Z distributed/pipeline/sync/test_copy.py::test_wait_multiple_tensors PASSED [100%]
2023-01-11T22:21:53.2385336Z 
2023-01-11T22:21:53.2385497Z ============================== 5 passed in 2.14s ===============================
2023-01-11T22:21:53.2385673Z 
2023-01-11T22:21:53.2385995Z ##[endgroup]
2023-01-11T22:21:53.2386632Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_copy (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_copy_6g3dkoff)
2023-01-11T22:21:53.2387011Z 
2023-01-11T22:21:53.2387306Z Running distributed/pipeline/sync/test_balance ... [2023-01-11 22:21:53.237529]
2023-01-11T22:21:53.2387898Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/test_balance.py', '-v'] ... [2023-01-11 22:21:53.237812]
2023-01-11T22:22:02.5527823Z 
2023-01-11T22:22:02.5528555Z Expand the folded group to see the log file of distributed/pipeline/sync/test_balance
2023-01-11T22:22:02.5529537Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/test_balance (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_balance_hp0i191t)
2023-01-11T22:22:02.5530105Z ============================= test session starts ==============================
2023-01-11T22:22:02.5530743Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:22:02.5531141Z cachedir: .pytest_cache
2023-01-11T22:22:02.5532034Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:22:02.5532485Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:22:02.5532816Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:22:02.5533410Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:22:02.5534024Z collecting ... collected 18 items
2023-01-11T22:22:02.5536520Z Running 18 items in this shard: test/distributed/pipeline/sync/test_balance.py::test_blockpartition, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions, test/distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda], test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale, test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu], test/distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu], test/distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda], test/distributed/pipeline/sync/test_balance.py::test_not_training, test/distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple, test/distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple, test/distributed/pipeline/sync/test_balance.py::test_already_has_grad
2023-01-11T22:22:02.5538754Z 
2023-01-11T22:22:02.5538966Z distributed/pipeline/sync/test_balance.py::test_blockpartition PASSED    [  5%]
2023-01-11T22:22:02.5539432Z distributed/pipeline/sync/test_balance.py::test_blockpartition_zeros PASSED [ 11%]
2023-01-11T22:22:02.5539937Z distributed/pipeline/sync/test_balance.py::test_blockpartition_non_positive_partitions PASSED [ 16%]
2023-01-11T22:22:02.5540445Z distributed/pipeline/sync/test_balance.py::test_blockpartition_short_sequence PASSED [ 22%]
2023-01-11T22:22:02.5540908Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cpu] SKIPPED [ 27%]
2023-01-11T22:22:02.5541371Z distributed/pipeline/sync/test_balance.py::test_balance_by_time[cuda] SKIPPED [ 33%]
2023-01-11T22:22:02.5541859Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_loop_resets_input PASSED [ 38%]
2023-01-11T22:22:02.5542335Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_latent PASSED [ 44%]
2023-01-11T22:22:02.5542775Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param PASSED [ 50%]
2023-01-11T22:22:02.5543242Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_param_scale PASSED [ 55%]
2023-01-11T22:22:02.5543711Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cpu] PASSED [ 61%]
2023-01-11T22:22:02.5544149Z distributed/pipeline/sync/test_balance.py::test_layerwise_sandbox[cuda] PASSED [ 66%]
2023-01-11T22:22:02.5544630Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cpu] PASSED [ 72%]
2023-01-11T22:22:02.5545112Z distributed/pipeline/sync/test_balance.py::test_sandbox_during_profiling[cuda] PASSED [ 77%]
2023-01-11T22:22:02.5545574Z distributed/pipeline/sync/test_balance.py::test_not_training PASSED      [ 83%]
2023-01-11T22:22:02.5546008Z distributed/pipeline/sync/test_balance.py::test_balance_by_time_tuple PASSED [ 88%]
2023-01-11T22:22:02.5546463Z distributed/pipeline/sync/test_balance.py::test_balance_by_size_tuple PASSED [ 94%]
2023-01-11T22:22:02.5546912Z distributed/pipeline/sync/test_balance.py::test_already_has_grad PASSED  [100%]
2023-01-11T22:22:02.5547162Z 
2023-01-11T22:22:02.5547315Z ======================== 16 passed, 2 skipped in 6.81s =========================
2023-01-11T22:22:02.5547519Z 
2023-01-11T22:22:02.5547839Z ##[endgroup]
2023-01-11T22:22:02.5548503Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/test_balance (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-test_balance_hp0i191t)
2023-01-11T22:22:02.5548892Z 
2023-01-11T22:22:02.5549203Z Running distributed/pipeline/sync/skip/test_stash_pop ... [2023-01-11 22:22:02.552917]
2023-01-11T22:22:02.5549824Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_stash_pop.py', '-v'] ... [2023-01-11 22:22:02.553202]
2023-01-11T22:22:04.9739027Z 
2023-01-11T22:22:04.9740033Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_stash_pop
2023-01-11T22:22:04.9742091Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_stash_pop (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_stash_pop_iv7les8a)
2023-01-11T22:22:04.9742674Z ============================= test session starts ==============================
2023-01-11T22:22:04.9743542Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:22:04.9744336Z cachedir: .pytest_cache
2023-01-11T22:22:04.9745394Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:22:04.9746285Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:22:04.9746924Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:22:04.9747660Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:22:04.9748047Z collecting ... collected 7 items
2023-01-11T22:22:04.9749036Z Running 7 items in this shard: test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed, test/distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none
2023-01-11T22:22:04.9749867Z 
2023-01-11T22:22:04.9750092Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash PASSED      [ 14%]
2023-01-11T22:22:04.9750532Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop PASSED        [ 28%]
2023-01-11T22:22:04.9750991Z distributed/pipeline/sync/skip/test_stash_pop.py::test_declare_but_not_use PASSED [ 42%]
2023-01-11T22:22:04.9751444Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_not_declared PASSED [ 57%]
2023-01-11T22:22:04.9751911Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_declared PASSED [ 71%]
2023-01-11T22:22:04.9752374Z distributed/pipeline/sync/skip/test_stash_pop.py::test_pop_not_stashed PASSED [ 85%]
2023-01-11T22:22:04.9752827Z distributed/pipeline/sync/skip/test_stash_pop.py::test_stash_none PASSED [100%]
2023-01-11T22:22:04.9753059Z 
2023-01-11T22:22:04.9753218Z ============================== 7 passed in 0.06s ===============================
2023-01-11T22:22:04.9753413Z 
2023-01-11T22:22:04.9753734Z ##[endgroup]
2023-01-11T22:22:04.9754423Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_stash_pop (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_stash_pop_iv7les8a)
2023-01-11T22:22:04.9754820Z 
2023-01-11T22:22:04.9755156Z Running distributed/pipeline/sync/skip/test_inspect_skip_layout ... [2023-01-11 22:22:04.974058]
2023-01-11T22:22:04.9755824Z Executing ['/opt/conda/bin/python', '-bb', '-m', 'pytest', 'distributed/pipeline/sync/skip/test_inspect_skip_layout.py', '-v'] ... [2023-01-11 22:22:04.974340]
2023-01-11T22:22:07.3045928Z 
2023-01-11T22:22:07.3046678Z Expand the folded group to see the log file of distributed/pipeline/sync/skip/test_inspect_skip_layout
2023-01-11T22:22:07.3047760Z ##[group]PRINTING LOG FILE of distributed/pipeline/sync/skip/test_inspect_skip_layout (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_inspect_skip_layout_26bjxmkl)
2023-01-11T22:22:07.3048355Z ============================= test session starts ==============================
2023-01-11T22:22:07.3048968Z platform linux -- Python 3.10.8, pytest-7.2.0, pluggy-1.0.0 -- /opt/conda/bin/python
2023-01-11T22:22:07.3049356Z cachedir: .pytest_cache
2023-01-11T22:22:07.3049945Z hypothesis profile 'default' -> database=DirectoryBasedExampleDatabase('/var/lib/jenkins/workspace/test/.hypothesis/examples')
2023-01-11T22:22:07.3050389Z torch: 2.0.0a0+git8419ddd
2023-01-11T22:22:07.3050707Z rootdir: /var/lib/jenkins/workspace, configfile: pytest.ini
2023-01-11T22:22:07.3051283Z plugins: hypothesis-5.35.1, flakefinder-1.1.0, rerunfailures-10.3, shard-0.1.2, xdist-3.1.0, xdoctest-1.1.0
2023-01-11T22:22:07.3051694Z collecting ... collected 6 items
2023-01-11T22:22:07.3052952Z Running 6 items in this shard: test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions, test/distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace
2023-01-11T22:22:07.3054001Z 
2023-01-11T22:22:07.3054260Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_no_skippables PASSED [ 16%]
2023-01-11T22:22:07.3054773Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_inner_partition PASSED [ 33%]
2023-01-11T22:22:07.3055312Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_adjoining_partitions PASSED [ 50%]
2023-01-11T22:22:07.3055846Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_far_partitions PASSED [ 66%]
2023-01-11T22:22:07.3056383Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_pop_2_from_different_partitions PASSED [ 83%]
2023-01-11T22:22:07.3056926Z distributed/pipeline/sync/skip/test_inspect_skip_layout.py::test_namespace PASSED [100%]
2023-01-11T22:22:07.3057222Z 
2023-01-11T22:22:07.3057389Z ============================== 6 passed in 0.04s ===============================
2023-01-11T22:22:07.3057588Z 
2023-01-11T22:22:07.3057900Z ##[endgroup]
2023-01-11T22:22:07.3058666Z FINISHED PRINTING LOG FILE of distributed/pipeline/sync/skip/test_inspect_skip_layout (/var/lib/jenkins/workspace/test/test-reports/distributed-pipeline-sync-skip-test_inspect_skip_layout_26bjxmkl)
2023-01-11T22:22:07.3059133Z 
2023-01-11T22:22:07.3059427Z Running distributed/elastic/timer/api_test ... [2023-01-11 22:22:07.304776]
2023-01-11T22:22:07.3060160Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:22:07.305069]
2023-01-11T22:22:09.1401131Z 
2023-01-11T22:22:09.1402346Z Expand the folded group to see the log file of distributed/elastic/timer/api_test
2023-01-11T22:22:09.1403798Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-api_test_u2808s1y)
2023-01-11T22:22:09.1404181Z 
2023-01-11T22:22:09.1405211Z ##[endgroup]
2023-01-11T22:22:09.1406594Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-api_test_u2808s1y)
2023-01-11T22:22:09.1406988Z 
2023-01-11T22:22:09.1407274Z Running distributed/_shard/test_sharder ... [2023-01-11 22:22:09.140203]
2023-01-11T22:22:09.1408862Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/test_sharder.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:22:09.140533]
2023-01-11T22:22:11.2585087Z 
2023-01-11T22:22:11.2585767Z Expand the folded group to see the log file of distributed/_shard/test_sharder
2023-01-11T22:22:11.2586671Z ##[group]PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_sharder_4zdkw5_k)
2023-01-11T22:22:11.2587502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_ejrtmgf
2023-01-11T22:22:11.2588067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_ejrtmgf/_remote_module_non_scriptable.py
2023-01-11T22:22:11.2588384Z 
2023-01-11T22:22:11.2588671Z ##[endgroup]
2023-01-11T22:22:11.2589369Z FINISHED PRINTING LOG FILE of distributed/_shard/test_sharder (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-test_sharder_4zdkw5_k)
2023-01-11T22:22:11.2589723Z 
2023-01-11T22:22:11.2590210Z Running distributed/_tools/test_memory_tracker ... [2023-01-11 22:22:11.258644]
2023-01-11T22:22:11.2592552Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tools/test_memory_tracker.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:22:11.258931]
2023-01-11T22:22:16.3780210Z 
2023-01-11T22:22:16.3781216Z Expand the folded group to see the log file of distributed/_tools/test_memory_tracker
2023-01-11T22:22:16.3783095Z ##[group]PRINTING LOG FILE of distributed/_tools/test_memory_tracker (/var/lib/jenkins/workspace/test/test-reports/distributed-_tools-test_memory_tracker_gwtm3ufu)
2023-01-11T22:22:16.3783503Z 
2023-01-11T22:22:16.3783603Z Running tests...
2023-01-11T22:22:16.3784266Z ----------------------------------------------------------------------
2023-01-11T22:22:16.3784905Z Test results will be stored in test-reports/python-unittest/distributed._tools.test_memory_tracker
2023-01-11T22:22:16.3785747Z   test_local_model (__main__.TestMemoryTracker)
2023-01-11T22:22:16.3786542Z Minimal test case to check the memory tracker can collect the expected ... ok (1.160s)
2023-01-11T22:22:16.3787283Z ------------------------------------------------
2023-01-11T22:22:16.3787917Z Top 20 ops that generates memory are:
2023-01-11T22:22:16.3788606Z 0.1.forward.cudnn_batch_norm.default_0: 24.5009765625MB
2023-01-11T22:22:16.3789053Z 0.0.forward.convolution.default_0: 24.5MB
2023-01-11T22:22:16.3789378Z 0.2.forward.relu.default_0: 24.5MB
2023-01-11T22:22:16.3789689Z 2.1.forward.div.Scalar_0: 24.5MB
2023-01-11T22:22:16.3790007Z 2.1.forward.threshold_backward.default_1: 24.49951171875MB
2023-01-11T22:22:16.3790351Z 2.0.forward.addmm.default_0: 4.00048828125MB
2023-01-11T22:22:16.3790670Z 2.1.forward.mm.default_0: 4.00048828125MB
2023-01-11T22:22:16.3790976Z 2.1.forward.nll_loss_forward.default_0: 0.0009765625MB
2023-01-11T22:22:16.3791289Z ._to_copy.default_0: 0.00048828125MB
2023-01-11T22:22:16.3791601Z 2.1.forward._log_softmax.default_0: 0.00048828125MB
2023-01-11T22:22:16.3791910Z 2.1.forward.ones_like.default_0: 0.00048828125MB
2023-01-11T22:22:16.3792269Z 2.1.forward.nll_loss_backward.default_0: 0.00048828125MB
2023-01-11T22:22:16.3792595Z 2.1.forward.mm.default_1: 0.00048828125MB
2023-01-11T22:22:16.3792904Z 2.1.forward.sum.dim_IntList_0: 0.00048828125MB
2023-01-11T22:22:16.3793181Z .lift_fresh.default_0: 0.0MB
2023-01-11T22:22:16.3793462Z 0.1.forward.add_.Tensor_0: 0.0MB
2023-01-11T22:22:16.3793749Z 1.forward.view.default_0: 0.0MB
2023-01-11T22:22:16.3794018Z 2.0.forward.t.default_0: 0.0MB
2023-01-11T22:22:16.3794308Z 2.1.forward.relu_.default_0: 0.0MB
2023-01-11T22:22:16.3794634Z 2.1.forward._log_softmax_backward_data.default_0: 0.0MB
2023-01-11T22:22:16.3795040Z ------------------------------------------------
2023-01-11T22:22:16.3795244Z 
2023-01-11T22:22:16.3795506Z ----------------------------------------------------------------------
2023-01-11T22:22:16.3795838Z Ran 1 test in 1.160s
2023-01-11T22:22:16.3795999Z 
2023-01-11T22:22:16.3796075Z OK
2023-01-11T22:22:16.3796207Z 
2023-01-11T22:22:16.3796333Z Generating XML reports...
2023-01-11T22:22:16.3796951Z Generated XML report: test-reports/python-unittest/distributed._tools.test_memory_tracker/TEST-TestMemoryTracker-20230111222214.xml
2023-01-11T22:22:16.3797308Z 
2023-01-11T22:22:16.3797622Z ##[endgroup]
2023-01-11T22:22:16.3798239Z FINISHED PRINTING LOG FILE of distributed/_tools/test_memory_tracker (/var/lib/jenkins/workspace/test/test-reports/distributed-_tools-test_memory_tracker_gwtm3ufu)
2023-01-11T22:22:16.3798601Z 
2023-01-11T22:22:16.3798884Z Running distributed/elastic/metrics/api_test ... [2023-01-11 22:22:16.378109]
2023-01-11T22:22:16.3799579Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/metrics/api_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:22:16.378409]
2023-01-11T22:22:20.2757785Z 
2023-01-11T22:22:20.2758678Z Expand the folded group to see the log file of distributed/elastic/metrics/api_test
2023-01-11T22:22:20.2760353Z ##[group]PRINTING LOG FILE of distributed/elastic/metrics/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-metrics-api_test_halfgjva)
2023-01-11T22:22:20.2761069Z 
2023-01-11T22:22:20.2761310Z Running tests...
2023-01-11T22:22:20.2762241Z ----------------------------------------------------------------------
2023-01-11T22:22:20.2762843Z Test results will be stored in test-reports/python-unittest/distributed.elastic.metrics.api_test
2023-01-11T22:22:20.2763568Z   test_get_metric_name (__main__.MetricsApiTest) ... ok (1.642s)
2023-01-11T22:22:20.2764780Z   test_inheritance (__main__.MetricsApiTest) ... ok (0.001s)
2023-01-11T22:22:20.2765512Z   test_profile (__main__.MetricsApiTest) ... ok (0.002s)
2023-01-11T22:22:20.2765898Z 
2023-01-11T22:22:20.2766197Z ----------------------------------------------------------------------
2023-01-11T22:22:20.2766513Z Ran 3 tests in 1.646s
2023-01-11T22:22:20.2766699Z 
2023-01-11T22:22:20.2766796Z OK
2023-01-11T22:22:20.2766932Z 
2023-01-11T22:22:20.2767060Z Generating XML reports...
2023-01-11T22:22:20.2767669Z Generated XML report: test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20230111222218.xml
2023-01-11T22:22:20.2768042Z 
2023-01-11T22:22:20.2768372Z ##[endgroup]
2023-01-11T22:22:20.2769002Z FINISHED PRINTING LOG FILE of distributed/elastic/metrics/api_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-metrics-api_test_halfgjva)
2023-01-11T22:22:20.2769380Z 
2023-01-11T22:22:20.2769651Z Running distributed/elastic/utils/logging_test ... [2023-01-11 22:22:20.275815]
2023-01-11T22:22:20.2770359Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/logging_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:22:20.276109]
2023-01-11T22:22:24.1465114Z 
2023-01-11T22:22:24.1465841Z Expand the folded group to see the log file of distributed/elastic/utils/logging_test
2023-01-11T22:22:24.1466851Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/logging_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-logging_test_givk_6xm)
2023-01-11T22:22:24.1467251Z 
2023-01-11T22:22:24.1467368Z Running tests...
2023-01-11T22:22:24.1467928Z ----------------------------------------------------------------------
2023-01-11T22:22:24.1468505Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.logging_test
2023-01-11T22:22:24.1468972Z   test_derive_module_name (__main__.LoggingTest) ... ok (1.616s)
2023-01-11T22:22:24.1469358Z   test_logger_name (__main__.LoggingTest) ... ok (0.002s)
2023-01-11T22:22:24.1469567Z 
2023-01-11T22:22:24.1469820Z ----------------------------------------------------------------------
2023-01-11T22:22:24.1470165Z Ran 2 tests in 1.619s
2023-01-11T22:22:24.1470331Z 
2023-01-11T22:22:24.1470427Z OK
2023-01-11T22:22:24.1470564Z 
2023-01-11T22:22:24.1470694Z Generating XML reports...
2023-01-11T22:22:24.1471290Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20230111222222.xml
2023-01-11T22:22:24.1471652Z 
2023-01-11T22:22:24.1471963Z ##[endgroup]
2023-01-11T22:22:24.1472582Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/logging_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-logging_test_givk_6xm)
2023-01-11T22:22:24.1472969Z 
2023-01-11T22:22:24.1473235Z Running distributed/test_launcher ... [2023-01-11 22:22:24.146601]
2023-01-11T22:22:24.1473897Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_launcher.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:22:24.146894]
2023-01-11T22:22:28.5234882Z 
2023-01-11T22:22:28.5235383Z Expand the folded group to see the log file of distributed/test_launcher
2023-01-11T22:22:28.5236339Z ##[group]PRINTING LOG FILE of distributed/test_launcher (/var/lib/jenkins/workspace/test/test-reports/distributed-test_launcher_zx6r9ws1)
2023-01-11T22:22:28.5236691Z 
2023-01-11T22:22:28.5236808Z Running tests...
2023-01-11T22:22:28.5237335Z ----------------------------------------------------------------------
2023-01-11T22:22:28.5237880Z Test results will be stored in test-reports/python-unittest/distributed.test_launcher
2023-01-11T22:22:28.5239290Z   test_launch_user_script (__main__.TestDistributedLaunch) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79488 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.624s)
2023-01-11T22:22:28.5239967Z 
2023-01-11T22:22:28.5240222Z ----------------------------------------------------------------------
2023-01-11T22:22:28.5240682Z Ran 1 test in 1.624s
2023-01-11T22:22:28.5240850Z 
2023-01-11T22:22:28.5240960Z OK (skipped=1)
2023-01-11T22:22:28.5241121Z 
2023-01-11T22:22:28.5241266Z Generating XML reports...
2023-01-11T22:22:28.5241857Z Generated XML report: test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20230111222226.xml
2023-01-11T22:22:28.5242214Z 
2023-01-11T22:22:28.5242535Z ##[endgroup]
2023-01-11T22:22:28.5243104Z FINISHED PRINTING LOG FILE of distributed/test_launcher (/var/lib/jenkins/workspace/test/test-reports/distributed-test_launcher_zx6r9ws1)
2023-01-11T22:22:28.5243422Z 
2023-01-11T22:22:28.5243704Z Running distributed/checkpoint/test_planner ... [2023-01-11 22:22:28.523590]
2023-01-11T22:22:28.5244718Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_planner.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:22:28.523895]
2023-01-11T22:22:32.4433331Z 
2023-01-11T22:22:32.4434145Z Expand the folded group to see the log file of distributed/checkpoint/test_planner
2023-01-11T22:22:32.4435213Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_planner (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_planner_ubgp82pa)
2023-01-11T22:22:32.4435723Z 
2023-01-11T22:22:32.4435920Z Running tests...
2023-01-11T22:22:32.4436733Z ----------------------------------------------------------------------
2023-01-11T22:22:32.4437439Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_planner
2023-01-11T22:22:32.4437897Z   test_global_plan (__main__.TestSavePlan) ... ok (1.629s)
2023-01-11T22:22:32.4438280Z   test_load_with_resharding (__main__.TestSavePlan) ... ok (0.004s)
2023-01-11T22:22:32.4438806Z   test_load_with_world_size_diff_by_one (__main__.TestSavePlan) ... ok (0.003s)
2023-01-11T22:22:32.4439392Z   test_local_load_plan (__main__.TestSavePlan) ... ok (0.003s)
2023-01-11T22:22:32.4439755Z   test_local_plan (__main__.TestSavePlan) ... ok (0.003s)
2023-01-11T22:22:32.4439974Z 
2023-01-11T22:22:32.4440259Z ----------------------------------------------------------------------
2023-01-11T22:22:32.4440571Z Ran 5 tests in 1.644s
2023-01-11T22:22:32.4440732Z 
2023-01-11T22:22:32.4440826Z OK
2023-01-11T22:22:32.4440978Z 
2023-01-11T22:22:32.4441105Z Generating XML reports...
2023-01-11T22:22:32.4441694Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_planner/TEST-TestSavePlan-20230111222230.xml
2023-01-11T22:22:32.4442050Z 
2023-01-11T22:22:32.4442375Z ##[endgroup]
2023-01-11T22:22:32.4443002Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_planner (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_planner_ubgp82pa)
2023-01-11T22:22:32.4443380Z 
2023-01-11T22:22:32.4443656Z Running distributed/fsdp/test_checkpoint_wrapper ... [2023-01-11 22:22:32.443424]
2023-01-11T22:22:32.4444615Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_checkpoint_wrapper.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:22:32.443717]
2023-01-11T22:22:37.4002019Z 
2023-01-11T22:22:37.4002531Z Expand the folded group to see the log file of distributed/fsdp/test_checkpoint_wrapper
2023-01-11T22:22:37.4003543Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_checkpoint_wrapper__lrdncqn)
2023-01-11T22:22:37.4003932Z 
2023-01-11T22:22:37.4004029Z Running tests...
2023-01-11T22:22:37.4004845Z ----------------------------------------------------------------------
2023-01-11T22:22:37.4005437Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper
2023-01-11T22:22:37.4005942Z   test_apply_activation_checkpointing (__main__.CheckpointWrapperTest)
2023-01-11T22:22:37.4006636Z Ensures that `apply_activation_checkpointing` can be used ... ok (1.755s)
2023-01-11T22:22:37.4007148Z   test_checkpoint_wrapper_cpu_offload (__main__.CheckpointWrapperTest) ... ok (0.411s)
2023-01-11T22:22:37.4007762Z   test_checkpoint_wrapper_kwarg_support (__main__.CheckpointWrapperTest) ... ok (0.009s)
2023-01-11T22:22:37.4008222Z   test_checkpoint_wrapper_parity (__main__.CheckpointWrapperTest)
2023-01-11T22:22:37.4008657Z Tests that using checkpoint_wrapper or the functional ... ok (0.524s)
2023-01-11T22:22:37.4009132Z   test_forward_missing_attributes (__main__.CheckpointWrapperTest) ... ok (0.001s)
2023-01-11T22:22:37.4009574Z   test_fqn (__main__.CheckpointWrapperTest) ... ok (0.001s)
2023-01-11T22:22:37.4010013Z   test_load_activation_checkpointed_module (__main__.CheckpointWrapperTest) ... ok (0.003s)
2023-01-11T22:22:37.4010301Z 
2023-01-11T22:22:37.4010599Z ----------------------------------------------------------------------
2023-01-11T22:22:37.4010952Z Ran 7 tests in 2.705s
2023-01-11T22:22:37.4011132Z 
2023-01-11T22:22:37.4011209Z OK
2023-01-11T22:22:37.4011352Z 
2023-01-11T22:22:37.4011483Z Generating XML reports...
2023-01-11T22:22:37.4012162Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20230111222234.xml
2023-01-11T22:22:37.4012572Z 
2023-01-11T22:22:37.4012933Z ##[endgroup]
2023-01-11T22:22:37.4013608Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_checkpoint_wrapper (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_checkpoint_wrapper__lrdncqn)
2023-01-11T22:22:37.4014030Z 
2023-01-11T22:22:37.4014374Z Running distributed/_shard/sharded_tensor/test_megatron_prototype ... [2023-01-11 22:22:37.400311]
2023-01-11T22:22:37.4015195Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/test_megatron_prototype.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:22:37.400651]
2023-01-11T22:22:43.7206538Z 
2023-01-11T22:22:43.7207082Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/test_megatron_prototype
2023-01-11T22:22:43.7208287Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_megatron_prototype (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_megatron_prototype_lxsmoz65)
2023-01-11T22:22:43.7208734Z 
2023-01-11T22:22:43.7208851Z Running tests...
2023-01-11T22:22:43.7209383Z ----------------------------------------------------------------------
2023-01-11T22:22:43.7209997Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype
2023-01-11T22:22:43.7210583Z   test_megatron_two_layer_prototype (__main__.TestShardedTensorMegatronLinear) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:22:43.7211120Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 55789
2023-01-11T22:22:43.7211580Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 55790
2023-01-11T22:22:43.7212020Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 55791
2023-01-11T22:22:43.7212472Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 55792
2023-01-11T22:22:43.7213200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:43.7213663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:43.7214247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:43.7214707Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:43.7215294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:43.7215752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:43.7216574Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:43.7217117Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:43.7217850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:43.7218337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:43.7218932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:43.7219440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:43.7220064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:43.7220531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:43.7221143Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:43.7221647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:43.7222123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:22:43.7222623Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:22:43.7223130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:22:43.7223639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:22:43.7224065Z skip: Need at least 4 CUDA devices (4.030s)
2023-01-11T22:22:43.7224254Z 
2023-01-11T22:22:43.7224543Z ----------------------------------------------------------------------
2023-01-11T22:22:43.7224894Z Ran 1 test in 4.030s
2023-01-11T22:22:43.7225066Z 
2023-01-11T22:22:43.7225180Z OK (skipped=1)
2023-01-11T22:22:43.7225344Z 
2023-01-11T22:22:43.7225454Z Generating XML reports...
2023-01-11T22:22:43.7226198Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20230111222239.xml
2023-01-11T22:22:43.7226658Z 
2023-01-11T22:22:43.7226999Z ##[endgroup]
2023-01-11T22:22:43.7227731Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/test_megatron_prototype (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-test_megatron_prototype_lxsmoz65)
2023-01-11T22:22:43.7228191Z 
2023-01-11T22:22:43.7228498Z Running distributed/elastic/utils/distributed_test ... [2023-01-11 22:22:43.720724]
2023-01-11T22:22:43.7229261Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/utils/distributed_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:22:43.721015]
2023-01-11T22:22:50.7699685Z 
2023-01-11T22:22:50.7700667Z Expand the folded group to see the log file of distributed/elastic/utils/distributed_test
2023-01-11T22:22:50.7702086Z ##[group]PRINTING LOG FILE of distributed/elastic/utils/distributed_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-distributed_test_b61jvl7q)
2023-01-11T22:22:50.7702508Z 
2023-01-11T22:22:50.7702655Z Running tests...
2023-01-11T22:22:50.7703436Z ----------------------------------------------------------------------
2023-01-11T22:22:50.7704067Z Test results will be stored in test-reports/python-unittest/distributed.elastic.utils.distributed_test
2023-01-11T22:22:50.7704569Z   test_create_store_multi (__main__.DistributedUtilTest) ... ok (1.639s)
2023-01-11T22:22:50.7705074Z   test_create_store_no_port_multi (__main__.DistributedUtilTest) ... ok (0.001s)
2023-01-11T22:22:50.7705805Z   test_create_store_single_server (__main__.DistributedUtilTest) ... ok (0.004s)
2023-01-11T22:22:50.7706244Z   test_create_store_timeout_on_server (__main__.DistributedUtilTest) ... ok (3.038s)
2023-01-11T22:22:50.7707067Z   test_create_store_timeout_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (c3943a31ca1f, 0).
2023-01-11T22:22:50.7707529Z ok (0.001s)
2023-01-11T22:22:50.7708187Z   test_port_already_in_use_on_server (__main__.DistributedUtilTest) ... [W socket.cpp:426] [c10d] The server socket has failed to bind to [::]:39689 (errno: 98 - Address already in use).
2023-01-11T22:22:50.7708995Z [W socket.cpp:426] [c10d] The server socket has failed to bind to 0.0.0.0:39689 (errno: 98 - Address already in use).
2023-01-11T22:22:50.7709471Z [E socket.cpp:462] [c10d] The server socket has failed to listen on any local network address.
2023-01-11T22:22:50.7709812Z ok (0.004s)
2023-01-11T22:22:50.7710263Z   test_port_already_in_use_on_worker (__main__.DistributedUtilTest) ... [E socket.cpp:860] [c10d] The client socket has timed out after 1s while trying to connect to (c3943a31ca1f, 40823).
2023-01-11T22:22:50.7710706Z ok (0.001s)
2023-01-11T22:22:50.7710856Z 
2023-01-11T22:22:50.7711132Z ----------------------------------------------------------------------
2023-01-11T22:22:50.7711448Z Ran 7 tests in 4.688s
2023-01-11T22:22:50.7711613Z 
2023-01-11T22:22:50.7711709Z OK
2023-01-11T22:22:50.7711843Z 
2023-01-11T22:22:50.7711973Z Generating XML reports...
2023-01-11T22:22:50.7712609Z Generated XML report: test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20230111222245.xml
2023-01-11T22:22:50.7713002Z 
2023-01-11T22:22:50.7713396Z ##[endgroup]
2023-01-11T22:22:50.7714062Z FINISHED PRINTING LOG FILE of distributed/elastic/utils/distributed_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-utils-distributed_test_b61jvl7q)
2023-01-11T22:22:50.7714462Z 
2023-01-11T22:22:50.7714786Z Running distributed/tensor/parallel/test_view_sharding_dim_change ... [2023-01-11 22:22:50.769992]
2023-01-11T22:22:50.7715555Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/tensor/parallel/test_view_sharding_dim_change.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:22:50.770284]
2023-01-11T22:22:57.8355054Z 
2023-01-11T22:22:57.8355908Z Expand the folded group to see the log file of distributed/tensor/parallel/test_view_sharding_dim_change
2023-01-11T22:22:57.8356995Z ##[group]PRINTING LOG FILE of distributed/tensor/parallel/test_view_sharding_dim_change (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_view_sharding_dim_change_s_cboznb)
2023-01-11T22:22:57.8357447Z 
2023-01-11T22:22:57.8357561Z Running tests...
2023-01-11T22:22:57.8358349Z ----------------------------------------------------------------------
2023-01-11T22:22:57.8358999Z Test results will be stored in test-reports/python-unittest/distributed.tensor.parallel.test_view_sharding_dim_change
2023-01-11T22:22:57.8359584Z   test_view_with_sharding_dim_change (__main__.TPViewShardingDimChangeTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:22:57.8360099Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56006
2023-01-11T22:22:57.8360849Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56007
2023-01-11T22:22:57.8361480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:57.8361944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:57.8362526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:57.8363002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:57.8363570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:22:57.8364052Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:22:57.8364907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:22:57.8365619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:22:57.8366066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:22:57.8366563Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:22:57.8367153Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:22:57.8367625Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:22:57.8368302Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:57.8369001Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:22:57.8369399Z ok (4.795s)
2023-01-11T22:22:57.8369549Z 
2023-01-11T22:22:57.8369801Z ----------------------------------------------------------------------
2023-01-11T22:22:57.8370134Z Ran 1 test in 4.795s
2023-01-11T22:22:57.8370297Z 
2023-01-11T22:22:57.8370391Z OK
2023-01-11T22:22:57.8370523Z 
2023-01-11T22:22:57.8370629Z Generating XML reports...
2023-01-11T22:22:57.8371342Z Generated XML report: test-reports/python-unittest/distributed.tensor.parallel.test_view_sharding_dim_change/TEST-TPViewShardingDimChangeTest-20230111222252.xml
2023-01-11T22:22:57.8371779Z 
2023-01-11T22:22:57.8372103Z ##[endgroup]
2023-01-11T22:22:57.8372819Z FINISHED PRINTING LOG FILE of distributed/tensor/parallel/test_view_sharding_dim_change (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_view_sharding_dim_change_s_cboznb)
2023-01-11T22:22:57.8373235Z 
2023-01-11T22:22:57.8373522Z Running distributed/elastic/timer/local_timer_test ... [2023-01-11 22:22:57.835553]
2023-01-11T22:22:57.8374229Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_test.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:22:57.835844]
2023-01-11T22:23:06.0383597Z 
2023-01-11T22:23:06.0384523Z Expand the folded group to see the log file of distributed/elastic/timer/local_timer_test
2023-01-11T22:23:06.0385540Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/local_timer_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_test_75y7b_x9)
2023-01-11T22:23:06.0385956Z 
2023-01-11T22:23:06.0386076Z Running tests...
2023-01-11T22:23:06.0386576Z ----------------------------------------------------------------------
2023-01-11T22:23:06.0387165Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_test
2023-01-11T22:23:06.0387630Z   test_acquire_release (__main__.LocalTimerServerTest)
2023-01-11T22:23:06.0387934Z tests that: ... ok (1.615s)
2023-01-11T22:23:06.0388287Z   test_expired_timers (__main__.LocalTimerServerTest)
2023-01-11T22:23:06.0388691Z tests that a single expired timer on a process should terminate ... ok (0.002s)
2023-01-11T22:23:06.0389088Z   test_valid_timers (__main__.LocalTimerServerTest)
2023-01-11T22:23:06.0389491Z tests that valid timers are processed correctly and the process is left alone ... ok (0.003s)
2023-01-11T22:23:06.0389915Z   test_watchdog_call_count (__main__.LocalTimerServerTest)
2023-01-11T22:23:06.0390409Z checks that the watchdog function ran wait/interval +- 1 times ... ok (0.103s)
2023-01-11T22:23:06.0390796Z   test_watchdog_empty_queue (__main__.LocalTimerServerTest)
2023-01-11T22:23:06.0391181Z checks that the watchdog can run on an empty queue ... ok (0.011s)
2023-01-11T22:23:06.0391569Z   test_client_interaction (__main__.LocalTimerTest) ... ok (0.004s)
2023-01-11T22:23:06.0391967Z   test_exception_propagation (__main__.LocalTimerTest) ... ok (0.011s)
2023-01-11T22:23:06.0392319Z   test_get_timer_recursive (__main__.LocalTimerTest)
2023-01-11T22:23:06.0393250Z If a function acquires a countdown timer with default scope, ... /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:06.0393800Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:06.0394387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:06.0394944Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:06.0395257Z ok (2.325s)
2023-01-11T22:23:06.0395556Z   test_happy_path (__main__.LocalTimerTest) ... ok (0.103s)
2023-01-11T22:23:06.0395895Z   test_no_client (__main__.LocalTimerTest) ... ok (0.011s)
2023-01-11T22:23:06.0396240Z   test_timer (__main__.LocalTimerTest) ... ok (0.156s)
2023-01-11T22:23:06.0396630Z   test_get (__main__.MultiprocessingRequestQueueTest) ... ok (0.023s)
2023-01-11T22:23:06.0397047Z   test_get_less_than_size (__main__.MultiprocessingRequestQueueTest)
2023-01-11T22:23:06.0397411Z Tests slow producer. ... ok (0.516s)
2023-01-11T22:23:06.0397766Z   test_get_size (__main__.MultiprocessingRequestQueueTest)
2023-01-11T22:23:06.0398158Z Creates a "producer" process that enqueues ``n`` elements ... ok (0.920s)
2023-01-11T22:23:06.0398394Z 
2023-01-11T22:23:06.0398671Z ----------------------------------------------------------------------
2023-01-11T22:23:06.0399008Z Ran 14 tests in 5.808s
2023-01-11T22:23:06.0399173Z 
2023-01-11T22:23:06.0399267Z OK
2023-01-11T22:23:06.0399383Z 
2023-01-11T22:23:06.0399510Z Generating XML reports...
2023-01-11T22:23:06.0400154Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20230111222259.xml
2023-01-11T22:23:06.0400968Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20230111222259.xml
2023-01-11T22:23:06.0401824Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20230111222259.xml
2023-01-11T22:23:06.0402223Z 
2023-01-11T22:23:06.0402534Z ##[endgroup]
2023-01-11T22:23:06.0403182Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/local_timer_test (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_test_75y7b_x9)
2023-01-11T22:23:06.0403568Z 
2023-01-11T22:23:06.0403886Z Running distributed/_shard/sharded_tensor/ops/test_embedding_bag ... [2023-01-11 22:23:06.038462]
2023-01-11T22:23:06.0405040Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_embedding_bag.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:23:06.038756]
2023-01-11T22:23:14.6374955Z 
2023-01-11T22:23:14.6375820Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_embedding_bag
2023-01-11T22:23:14.6376913Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding_bag (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_bag_e_8ukwgg)
2023-01-11T22:23:14.6377721Z 
2023-01-11T22:23:14.6377935Z Running tests...
2023-01-11T22:23:14.6378557Z ----------------------------------------------------------------------
2023-01-11T22:23:14.6379170Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag
2023-01-11T22:23:14.6379796Z   test_sharded_embedding_bag_colwise (__main__.TestShardedEmbeddingBag) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:23:14.6380286Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56210
2023-01-11T22:23:14.6380744Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56211
2023-01-11T22:23:14.6381193Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56212
2023-01-11T22:23:14.6381623Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56213
2023-01-11T22:23:14.6382263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:14.6382979Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:14.6383586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:14.6384153Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:14.6384741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:14.6385190Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:14.6385766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:14.6386219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:14.6386801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:14.6387246Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:14.6387802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:14.6388269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:14.6388848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:14.6389291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:14.6389848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:14.6390315Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:14.6390755Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:14.6391214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:23:14.6391688Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:23:14.6392146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:14.6392543Z skip: Need at least 4 CUDA devices (4.015s)
2023-01-11T22:23:14.6393035Z   test_sharded_embedding_bag_rowwise (__main__.TestShardedEmbeddingBag) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56346
2023-01-11T22:23:14.6393589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56347
2023-01-11T22:23:14.6394038Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56348
2023-01-11T22:23:14.6394465Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56349
2023-01-11T22:23:14.6395086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:14.6395534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:14.6396111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:14.6396567Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:14.6397152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:14.6397597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:14.6398168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:14.6398620Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:14.6399200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:14.6399643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:14.6400262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:14.6400736Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:14.6401370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:14.6401812Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:14.6402365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:14.6402835Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:14.6403273Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:14.6403732Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:23:14.6404426Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:14.6404907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:23:14.6405305Z skip: Need at least 4 CUDA devices (2.309s)
2023-01-11T22:23:14.6405504Z 
2023-01-11T22:23:14.6405768Z ----------------------------------------------------------------------
2023-01-11T22:23:14.6406097Z Ran 2 tests in 6.325s
2023-01-11T22:23:14.6406261Z 
2023-01-11T22:23:14.6406371Z OK (skipped=2)
2023-01-11T22:23:14.6406526Z 
2023-01-11T22:23:14.6406650Z Generating XML reports...
2023-01-11T22:23:14.6407303Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20230111222307.xml
2023-01-11T22:23:14.6407708Z 
2023-01-11T22:23:14.6408033Z ##[endgroup]
2023-01-11T22:23:14.6408730Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_embedding_bag (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_embedding_bag_e_8ukwgg)
2023-01-11T22:23:14.6409129Z 
2023-01-11T22:23:14.6409435Z Running distributed/_shard/sharded_tensor/ops/test_softmax ... [2023-01-11 22:23:14.637538]
2023-01-11T22:23:14.6410176Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_shard/sharded_tensor/ops/test_softmax.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:23:14.637847]
2023-01-11T22:23:23.3598001Z 
2023-01-11T22:23:23.3598736Z Expand the folded group to see the log file of distributed/_shard/sharded_tensor/ops/test_softmax
2023-01-11T22:23:23.3599743Z ##[group]PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_softmax (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_softmax_8inlkw17)
2023-01-11T22:23:23.3600411Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiepfwccx
2023-01-11T22:23:23.3600972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiepfwccx/_remote_module_non_scriptable.py
2023-01-11T22:23:23.3601581Z 
2023-01-11T22:23:23.3601725Z Running tests...
2023-01-11T22:23:23.3602413Z ----------------------------------------------------------------------
2023-01-11T22:23:23.3603009Z Test results will be stored in test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax
2023-01-11T22:23:23.3603545Z   test_sharded_softmax_basic (__main__.TestShardedSoftmax) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:23:23.3604569Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56517
2023-01-11T22:23:23.3605070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56518
2023-01-11T22:23:23.3605502Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56519
2023-01-11T22:23:23.3605945Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56520
2023-01-11T22:23:23.3606593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:23.3607278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:23.3607895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:23.3608473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:23.3609063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:23.3609492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:23.3610066Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:23.3610530Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:23.3611107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:23.3611538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:23.3612115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:23.3612585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:23.3613146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:23.3613594Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:23.3614168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:23.3614634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:23.3615131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq5gw4sdx
2023-01-11T22:23:23.3615684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq5gw4sdx/_remote_module_non_scriptable.py
2023-01-11T22:23:23.3616201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:23.3616688Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprpcdqy5z
2023-01-11T22:23:23.3617235Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprpcdqy5z/_remote_module_non_scriptable.py
2023-01-11T22:23:23.3617769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92ribh3o
2023-01-11T22:23:23.3618307Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92ribh3o/_remote_module_non_scriptable.py
2023-01-11T22:23:23.3618818Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpje252dv6
2023-01-11T22:23:23.3619351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpje252dv6/_remote_module_non_scriptable.py
2023-01-11T22:23:23.3619862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:23:23.3620341Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:23:23.3620793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:23.3621185Z skip: Need at least 4 CUDA devices (4.038s)
2023-01-11T22:23:23.3621678Z   test_sharded_softmax_on_sharding_dim (__main__.TestShardedSoftmax) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56653
2023-01-11T22:23:23.3622204Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56654
2023-01-11T22:23:23.3622654Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56655
2023-01-11T22:23:23.3623094Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56656
2023-01-11T22:23:23.3623713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:23.3624213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:23.3624805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:23.3625332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:23.3625901Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:23.3626350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:23.3626922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:23.3627390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:23.3627953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:23.3628401Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:23.3628979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:23.3629446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:23.3630009Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:23.3630454Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:23.3631024Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:23.3631470Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:23.3631934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9jfkke8z
2023-01-11T22:23:23.3632478Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9jfkke8z/_remote_module_non_scriptable.py
2023-01-11T22:23:23.3633013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpceilijuf
2023-01-11T22:23:23.3633532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpceilijuf/_remote_module_non_scriptable.py
2023-01-11T22:23:23.3634068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjwucwg_8
2023-01-11T22:23:23.3634607Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjwucwg_8/_remote_module_non_scriptable.py
2023-01-11T22:23:23.3635128Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk44bt5gl
2023-01-11T22:23:23.3635662Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk44bt5gl/_remote_module_non_scriptable.py
2023-01-11T22:23:23.3636172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:23.3636647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:23:23.3637102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:23.3637567Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:23:23.3637962Z skip: Need at least 4 CUDA devices (2.409s)
2023-01-11T22:23:23.3638159Z 
2023-01-11T22:23:23.3638432Z ----------------------------------------------------------------------
2023-01-11T22:23:23.3638744Z Ran 2 tests in 6.447s
2023-01-11T22:23:23.3638907Z 
2023-01-11T22:23:23.3639016Z OK (skipped=2)
2023-01-11T22:23:23.3639171Z 
2023-01-11T22:23:23.3639295Z Generating XML reports...
2023-01-11T22:23:23.3639911Z Generated XML report: test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20230111222316.xml
2023-01-11T22:23:23.3640289Z 
2023-01-11T22:23:23.3640619Z ##[endgroup]
2023-01-11T22:23:23.3641292Z FINISHED PRINTING LOG FILE of distributed/_shard/sharded_tensor/ops/test_softmax (/var/lib/jenkins/workspace/test/test-reports/distributed-_shard-sharded_tensor-ops-test_softmax_8inlkw17)
2023-01-11T22:23:23.3641755Z 
2023-01-11T22:23:23.3642044Z Running distributed/_tensor/test_view_ops ... [2023-01-11 22:23:23.359943]
2023-01-11T22:23:23.3642710Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_view_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:23:23.360222]
2023-01-11T22:23:32.3388425Z 
2023-01-11T22:23:32.3388932Z Expand the folded group to see the log file of distributed/_tensor/test_view_ops
2023-01-11T22:23:32.3390433Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_view_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_view_ops_h9up9s6y)
2023-01-11T22:23:32.3390907Z 
2023-01-11T22:23:32.3391113Z Running tests...
2023-01-11T22:23:32.3392034Z ----------------------------------------------------------------------
2023-01-11T22:23:32.3393171Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_view_ops
2023-01-11T22:23:32.3394067Z   test_view_groups (__main__.TestViewOps) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:23:32.3395070Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 56824
2023-01-11T22:23:32.3395656Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 56825
2023-01-11T22:23:32.3396113Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 56826
2023-01-11T22:23:32.3396537Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 56827
2023-01-11T22:23:32.3397284Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 56828
2023-01-11T22:23:32.3398201Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 56829
2023-01-11T22:23:32.3399251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:32.3399718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:32.3400308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:32.3400789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:32.3401358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:32.3401857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:32.3402433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:32.3402907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:32.3403468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:32.3403917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:32.3404806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:32.3405283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:32.3405849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:32.3406306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:32.3406880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:32.3407346Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:32.3407906Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:32.3408353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:32.3408929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:32.3409628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:32.3410232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:32.3410775Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:32.3411354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:32.3411800Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:32.3412366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T22:23:32.3413213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:23:32.3414002Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:23:32.3414773Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T22:23:32.3415856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:32.3416910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:32.3417601Z ok (4.125s)
2023-01-11T22:23:32.3418413Z   test_view_ops (__main__.TestViewOps) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57028
2023-01-11T22:23:32.3418952Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 57029
2023-01-11T22:23:32.3419391Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 57030
2023-01-11T22:23:32.3419843Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 57031
2023-01-11T22:23:32.3420285Z INFO:torch.testing._internal.common_distributed:Started process 4 with pid 57032
2023-01-11T22:23:32.3420731Z INFO:torch.testing._internal.common_distributed:Started process 5 with pid 57033
2023-01-11T22:23:32.3421374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:32.3421835Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:32.3422418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:32.3422894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:32.3423456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:32.3423906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:32.3424485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:32.3424942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:32.3425531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:32.3425982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:32.3426560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:32.3427012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:32.3427590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:32.3428034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:32.3428588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:32.3429056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:32.3429746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:32.3430208Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:32.3430766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:32.3431299Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:32.3431883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:32.3432330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:32.3432883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:32.3433351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:32.3433792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:23:32.3434258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:23:32.3434723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 4
2023-01-11T22:23:32.3435197Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:23:32.3435667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 5
2023-01-11T22:23:32.3436115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:32.3436502Z skip: Need at least 6 CUDA devices (2.520s)
2023-01-11T22:23:32.3436700Z 
2023-01-11T22:23:32.3436978Z ----------------------------------------------------------------------
2023-01-11T22:23:32.3437291Z Ran 2 tests in 6.645s
2023-01-11T22:23:32.3437456Z 
2023-01-11T22:23:32.3437563Z OK (skipped=1)
2023-01-11T22:23:32.3437717Z 
2023-01-11T22:23:32.3437840Z Generating XML reports...
2023-01-11T22:23:32.3438421Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_view_ops/TEST-TestViewOps-20230111222325.xml
2023-01-11T22:23:32.3438735Z 
2023-01-11T22:23:32.3439080Z ##[endgroup]
2023-01-11T22:23:32.3439682Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_view_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_view_ops_h9up9s6y)
2023-01-11T22:23:32.3440028Z 
2023-01-11T22:23:32.3440294Z Running distributed/fsdp/test_fsdp_input ... [2023-01-11 22:23:32.339023]
2023-01-11T22:23:32.3440957Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_input.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:23:32.339308]
2023-01-11T22:23:43.3084326Z 
2023-01-11T22:23:43.3085425Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_input
2023-01-11T22:23:43.3087250Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_input (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_input_wv3n96ps)
2023-01-11T22:23:43.3087696Z 
2023-01-11T22:23:43.3087816Z Running tests...
2023-01-11T22:23:43.3088345Z ----------------------------------------------------------------------
2023-01-11T22:23:43.3088907Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_input
2023-01-11T22:23:43.3089336Z   test_input_type_dict (__main__.TestInput)
2023-01-11T22:23:43.3089743Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:23:43.3090228Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57267
2023-01-11T22:23:43.3090861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:43.3091304Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:43.3091886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:43.3092606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:43.3093093Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:43.3093850Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:23:43.3094381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:43.3095173Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2023-01-11T22:23:43.3095664Z   warnings.warn(
2023-01-11T22:23:43.3096820Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:23:43.3097621Z   warnings.warn(
2023-01-11T22:23:43.3097871Z dist init r=0, world=1
2023-01-11T22:23:43.3098114Z ok (5.129s)
2023-01-11T22:23:43.3098370Z   test_input_type_list (__main__.TestInput)
2023-01-11T22:23:43.3098848Z Test FSDP with input being a list or a dict, only single GPU. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 57309
2023-01-11T22:23:43.3099539Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:43.3099996Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:43.3100562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:43.3101037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:43.3101498Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:23:43.3102150Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:23:43.3102684Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:23:43.3103468Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2023-01-11T22:23:43.3103959Z   warnings.warn(
2023-01-11T22:23:43.3105113Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:23:43.3105901Z   warnings.warn(
2023-01-11T22:23:43.3106155Z dist init r=0, world=1
2023-01-11T22:23:43.3106396Z ok (3.509s)
2023-01-11T22:23:43.3106543Z 
2023-01-11T22:23:43.3106800Z ----------------------------------------------------------------------
2023-01-11T22:23:43.3107133Z Ran 2 tests in 8.638s
2023-01-11T22:23:43.3107294Z 
2023-01-11T22:23:43.3107388Z OK
2023-01-11T22:23:43.3107520Z 
2023-01-11T22:23:43.3107625Z Generating XML reports...
2023-01-11T22:23:43.3108193Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20230111222334.xml
2023-01-11T22:23:43.3108526Z 
2023-01-11T22:23:43.3108852Z ##[endgroup]
2023-01-11T22:23:43.3109503Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_input (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_input_wv3n96ps)
2023-01-11T22:23:43.3109871Z 
2023-01-11T22:23:43.3110177Z Running distributed/elastic/timer/local_timer_example ... [2023-01-11 22:23:43.308519]
2023-01-11T22:23:43.3110956Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/elastic/timer/local_timer_example.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:23:43.308816]
2023-01-11T22:23:56.6660460Z 
2023-01-11T22:23:56.6661226Z Expand the folded group to see the log file of distributed/elastic/timer/local_timer_example
2023-01-11T22:23:56.6662245Z ##[group]PRINTING LOG FILE of distributed/elastic/timer/local_timer_example (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_example_nq_32ob1)
2023-01-11T22:23:56.6662736Z 
2023-01-11T22:23:56.6662934Z Running tests...
2023-01-11T22:23:56.6663767Z ----------------------------------------------------------------------
2023-01-11T22:23:56.6664382Z Test results will be stored in test-reports/python-unittest/distributed.elastic.timer.local_timer_example
2023-01-11T22:23:56.6665580Z   test_example_start_method_spawn (__main__.LocalTimerExample) ... [INFO] 2023-01-11 22:23:46,765 driver: init
2023-01-11T22:23:56.6666478Z [INFO] 2023-01-11 22:23:46,784 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2023-01-11T22:23:56.6666938Z [INFO] 2023-01-11 22:23:46,784 api: Starting watchdog thread...
2023-01-11T22:23:56.6667501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6667961Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6668540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6668999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6669580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6670043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6670617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6671077Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6671659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6672105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6672659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6673125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6673702Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6674151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6674711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6675182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6675754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6676185Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6676756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6677222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6677870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6678321Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6679153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6679708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6680297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6680745Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6681319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6681771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6682350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6682793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6683353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6683819Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6684592Z [INFO] 2023-01-11 22:23:48,691 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6685108Z [INFO] 2023-01-11 22:23:48,692 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6685576Z [INFO] 2023-01-11 22:23:48,717 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6686053Z [INFO] 2023-01-11 22:23:48,721 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6686538Z [INFO] 2023-01-11 22:23:48,765 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6687002Z [INFO] 2023-01-11 22:23:48,768 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6687477Z [INFO] 2023-01-11 22:23:48,815 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6687950Z [INFO] 2023-01-11 22:23:48,818 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6688525Z [INFO] 2023-01-11 22:23:49,832 api: Reaping worker_id=[57387]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:23:56.6689027Z [INFO] 2023-01-11 22:23:49,833 api: Successfully reaped worker=[57387]
2023-01-11T22:23:56.6689595Z [INFO] 2023-01-11 22:23:49,833 api: Reaping worker_id=[57389]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:23:56.6690102Z [INFO] 2023-01-11 22:23:49,833 api: Successfully reaped worker=[57389]
2023-01-11T22:23:56.6690635Z [INFO] 2023-01-11 22:23:49,874 api: Reaping worker_id=[57393]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:23:56.6691136Z [INFO] 2023-01-11 22:23:49,875 api: Successfully reaped worker=[57393]
2023-01-11T22:23:56.6691693Z [INFO] 2023-01-11 22:23:49,915 api: Reaping worker_id=[57391]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:23:56.6692203Z [INFO] 2023-01-11 22:23:49,916 api: Successfully reaped worker=[57391]
2023-01-11T22:23:56.6692622Z [INFO] 2023-01-11 22:23:49,926 api: Stopping LocalTimerServer
2023-01-11T22:23:56.6693047Z [INFO] 2023-01-11 22:23:49,927 api: Stopping watchdog thread...
2023-01-11T22:23:56.6693341Z ok (4.794s)
2023-01-11T22:23:56.6693889Z   test_torch_mp_example (__main__.LocalTimerExample) ... [INFO] 2023-01-11 22:23:49,938 api: Starting LocalTimerServer... max_interval=0.01, daemon=True
2023-01-11T22:23:56.6694429Z [INFO] 2023-01-11 22:23:49,938 api: Starting watchdog thread...
2023-01-11T22:23:56.6694985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6695440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6695998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6696566Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6697166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6697687Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6698246Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6698711Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6699291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6699716Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6700285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6700750Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6701333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6701758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6702337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6702801Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6703356Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6703799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6704375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6704840Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6705401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6705849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6706423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6706887Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6707444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6707887Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6708456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6708901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6709480Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6709923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6710492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6710942Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6711431Z [INFO] 2023-01-11 22:23:51,957 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6711913Z [INFO] 2023-01-11 22:23:51,961 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6712374Z [INFO] 2023-01-11 22:23:51,965 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6712858Z [INFO] 2023-01-11 22:23:52,021 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6713335Z [INFO] 2023-01-11 22:23:52,039 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6713873Z [INFO] 2023-01-11 22:23:52,042 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6714341Z [INFO] 2023-01-11 22:23:52,050 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6714808Z [INFO] 2023-01-11 22:23:52,054 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6715442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6715875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6716505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6716977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6717559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6717985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6718562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6719028Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6719603Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6720029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6720602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6721064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6721624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6722068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6722643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6723106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6723659Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6724109Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6724939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6725385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6725957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6726403Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6726974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6727428Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6728007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6728453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6729019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6729462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6730040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:23:56.6730483Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:23:56.6731036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:23:56.6731585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:23:56.6732092Z [INFO] 2023-01-11 22:23:55,071 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6732649Z [INFO] 2023-01-11 22:23:55,073 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6733113Z [INFO] 2023-01-11 22:23:55,089 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6733580Z [INFO] 2023-01-11 22:23:55,089 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6734053Z [INFO] 2023-01-11 22:23:55,092 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6734508Z [INFO] 2023-01-11 22:23:55,106 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6734978Z [INFO] 2023-01-11 22:23:55,121 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6735448Z [INFO] 2023-01-11 22:23:55,124 api: Timer client configured to: LocalTimerClient
2023-01-11T22:23:56.6736021Z [INFO] 2023-01-11 22:23:56,191 api: Reaping worker_id=[57937]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:23:56.6736519Z [INFO] 2023-01-11 22:23:56,191 api: Successfully reaped worker=[57937]
2023-01-11T22:23:56.6737083Z [INFO] 2023-01-11 22:23:56,191 api: Reaping worker_id=[57934]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:23:56.6737588Z [INFO] 2023-01-11 22:23:56,192 api: Successfully reaped worker=[57934]
2023-01-11T22:23:56.6738140Z [INFO] 2023-01-11 22:23:56,202 api: Reaping worker_id=[57933]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:23:56.6738624Z [INFO] 2023-01-11 22:23:56,202 api: Successfully reaped worker=[57933]
2023-01-11T22:23:56.6739182Z [INFO] 2023-01-11 22:23:56,213 api: Reaping worker_id=[57935]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:23:56.6739692Z [INFO] 2023-01-11 22:23:56,213 api: Successfully reaped worker=[57935]
2023-01-11T22:23:56.6740227Z [INFO] 2023-01-11 22:23:56,213 api: Reaping worker_id=[57932]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:23:56.6740725Z [INFO] 2023-01-11 22:23:56,213 api: Successfully reaped worker=[57932]
2023-01-11T22:23:56.6741285Z [INFO] 2023-01-11 22:23:56,223 api: Reaping worker_id=[57939]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:23:56.6741792Z [INFO] 2023-01-11 22:23:56,224 api: Successfully reaped worker=[57939]
2023-01-11T22:23:56.6742325Z [INFO] 2023-01-11 22:23:56,234 api: Reaping worker_id=[57938]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:23:56.6742832Z [INFO] 2023-01-11 22:23:56,234 api: Successfully reaped worker=[57938]
2023-01-11T22:23:56.6743391Z [INFO] 2023-01-11 22:23:56,234 api: Reaping worker_id=[57936]. Expired timers: ['/opt/conda/lib/python3.10/contextlib.py#135']
2023-01-11T22:23:56.6743896Z [INFO] 2023-01-11 22:23:56,235 api: Successfully reaped worker=[57936]
2023-01-11T22:23:56.6744310Z [INFO] 2023-01-11 22:23:56,247 api: Stopping LocalTimerServer
2023-01-11T22:23:56.6744740Z [INFO] 2023-01-11 22:23:56,247 api: Stopping watchdog thread...
2023-01-11T22:23:56.6745028Z ok (6.318s)
2023-01-11T22:23:56.6745161Z 
2023-01-11T22:23:56.6745436Z ----------------------------------------------------------------------
2023-01-11T22:23:56.6745764Z Ran 2 tests in 11.113s
2023-01-11T22:23:56.6745929Z 
2023-01-11T22:23:56.6746022Z OK
2023-01-11T22:23:56.6746154Z 
2023-01-11T22:23:56.6746260Z Generating XML reports...
2023-01-11T22:23:56.6746889Z Generated XML report: test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20230111222345.xml
2023-01-11T22:23:56.6747271Z 
2023-01-11T22:23:56.6747606Z ##[endgroup]
2023-01-11T22:23:56.6748251Z FINISHED PRINTING LOG FILE of distributed/elastic/timer/local_timer_example (/var/lib/jenkins/workspace/test/test-reports/distributed-elastic-timer-local_timer_example_nq_32ob1)
2023-01-11T22:23:56.6748654Z 
2023-01-11T22:23:56.6748984Z Running distributed/_tensor/test_math_ops ... [2023-01-11 22:23:56.666168]
2023-01-11T22:23:56.6749673Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_math_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:23:56.666448]
2023-01-11T22:24:10.6142925Z 
2023-01-11T22:24:10.6143654Z Expand the folded group to see the log file of distributed/_tensor/test_math_ops
2023-01-11T22:24:10.6145125Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_math_ops_by5sclu0)
2023-01-11T22:24:10.6145495Z 
2023-01-11T22:24:10.6145613Z Running tests...
2023-01-11T22:24:10.6146160Z ----------------------------------------------------------------------
2023-01-11T22:24:10.6146698Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_math_ops
2023-01-11T22:24:10.6147220Z   test_softmax_fwd (__main__.DistMathOpsTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:24:10.6148107Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58239
2023-01-11T22:24:10.6148807Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58240
2023-01-11T22:24:10.6150237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:10.6150836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:10.6151429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:10.6151896Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:10.6152729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:10.6153560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:10.6154809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:10.6155294Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:10.6155776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:10.6156283Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:24:10.6156780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:10.6157251Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:24:10.6157931Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:10.6158632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:10.6159015Z ok (4.919s)
2023-01-11T22:24:10.6159444Z   test_softmax_with_bwd (__main__.DistMathOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58320
2023-01-11T22:24:10.6159961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58321
2023-01-11T22:24:10.6160580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:10.6161016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:10.6161598Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:10.6162072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:10.6162652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:10.6163081Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:10.6163919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:10.6164778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:10.6165204Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:10.6165813Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:24:10.6166304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:10.6166809Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:24:10.6167471Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:10.6168171Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:10.6168567Z ok (3.411s)
2023-01-11T22:24:10.6168964Z   test_sum (__main__.DistMathOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58405
2023-01-11T22:24:10.6169467Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58406
2023-01-11T22:24:10.6170086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:10.6170543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:10.6171107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:10.6171577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:10.6172159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:10.6172610Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:10.6173168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:10.6173635Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:10.6174078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:10.6174558Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:24:10.6175049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:10.6175540Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:24:10.6176208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:10.6176886Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:10.6177287Z ok (3.310s)
2023-01-11T22:24:10.6177437Z 
2023-01-11T22:24:10.6177707Z ----------------------------------------------------------------------
2023-01-11T22:24:10.6178040Z Ran 3 tests in 11.640s
2023-01-11T22:24:10.6178190Z 
2023-01-11T22:24:10.6178287Z OK
2023-01-11T22:24:10.6178421Z 
2023-01-11T22:24:10.6178546Z Generating XML reports...
2023-01-11T22:24:10.6179132Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_math_ops/TEST-DistMathOpsTest-20230111222358.xml
2023-01-11T22:24:10.6179474Z 
2023-01-11T22:24:10.6179804Z ##[endgroup]
2023-01-11T22:24:10.6180406Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_math_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_math_ops_by5sclu0)
2023-01-11T22:24:10.6180757Z 
2023-01-11T22:24:10.6181028Z Running distributed/fsdp/test_fsdp_apply ... [2023-01-11 22:24:10.614362]
2023-01-11T22:24:10.6181782Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_apply.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:24:10.614656]
2023-01-11T22:24:25.1426357Z 
2023-01-11T22:24:25.1426833Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_apply
2023-01-11T22:24:25.1428425Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_apply_u5cypieg)
2023-01-11T22:24:25.1428798Z 
2023-01-11T22:24:25.1428911Z Running tests...
2023-01-11T22:24:25.1429410Z ----------------------------------------------------------------------
2023-01-11T22:24:25.1429974Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_apply
2023-01-11T22:24:25.1430763Z   test_apply_in_summon_raises_error (__main__.TestApply)
2023-01-11T22:24:25.1431238Z Tests that calling ``apply()`` on an FSDP instance inside the ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:24:25.1431704Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58521
2023-01-11T22:24:25.1432175Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58522
2023-01-11T22:24:25.1432816Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:25.1433279Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:25.1433840Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:25.1434317Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:25.1434897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:25.1435429Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:25.1435986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:25.1436454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:25.1436906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:24:25.1437391Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:24:25.1438057Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:25.1438751Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:25.1439277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:25.1439736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:25.1441026Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:24:25.1441815Z   warnings.warn(
2023-01-11T22:24:25.1442977Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:24:25.1443756Z   warnings.warn(
2023-01-11T22:24:25.1444003Z   File "<string>", line 1, in <module>
2023-01-11T22:24:25.1444792Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
2023-01-11T22:24:25.1445186Z     exitcode = _main(fd, parent_sentinel)
2023-01-11T22:24:25.1445559Z   File "/opt/conda/lib/python3.10/multiprocessing/spawn.py", line 129, in _main
2023-01-11T22:24:25.1445989Z     return self._bootstrap(parent_sentinel)
2023-01-11T22:24:25.1446381Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 314, in _bootstrap
2023-01-11T22:24:25.1446717Z     self.run()
2023-01-11T22:24:25.1447031Z   File "/opt/conda/lib/python3.10/multiprocessing/process.py", line 108, in run
2023-01-11T22:24:25.1447401Z     self._target(*self._args, **self._kwargs)
2023-01-11T22:24:25.1447933Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_fsdp.py", line 824, in _run
2023-01-11T22:24:25.1448312Z     self.run_test(test_name, pipe)
2023-01-11T22:24:25.1448840Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 658, in run_test
2023-01-11T22:24:25.1449237Z     getattr(self, test_name)()
2023-01-11T22:24:25.1449754Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 536, in wrapper
2023-01-11T22:24:25.1450108Z     fn()
2023-01-11T22:24:25.1450590Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_distributed.py", line 167, in wrapper
2023-01-11T22:24:25.1450977Z     return func(*args, **kwargs)
2023-01-11T22:24:25.1451383Z   File "/var/lib/jenkins/workspace/test/distributed/fsdp/test_fsdp_apply.py", line 98, in test_apply_in_summon_raises_error
2023-01-11T22:24:25.1451820Z     transformer.apply(self._init_linear_weights)
2023-01-11T22:24:25.1452383Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 491, in apply
2023-01-11T22:24:25.1452818Z     self._assert_state(TrainingState.IDLE)
2023-01-11T22:24:25.1453376Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/fully_sharded_data_parallel.py", line 930, in _assert_state
2023-01-11T22:24:25.1453788Z     traceback.print_stack()
2023-01-11T22:24:25.1454054Z dist init r=1, world=2
2023-01-11T22:24:25.1454286Z dist init r=0, world=2
2023-01-11T22:24:25.1454615Z Asserting FSDP instance is: FullyShardedDataParallel(
2023-01-11T22:24:25.1455005Z   (_fsdp_wrapped_module): TransformerWithSharedParams(
2023-01-11T22:24:25.1455331Z     (embed_tokens): Embedding(23, 16)
2023-01-11T22:24:25.1455614Z     (transformer): Transformer(
2023-01-11T22:24:25.1455902Z       (encoder): TransformerEncoder(
2023-01-11T22:24:25.1456168Z         (layers): ModuleList(
2023-01-11T22:24:25.1456533Z           (0-1): 2 x FullyShardedDataParallel(
2023-01-11T22:24:25.1456897Z             (_fsdp_wrapped_module): TransformerEncoderLayer(
2023-01-11T22:24:25.1457242Z               (self_attn): MultiheadAttention(
2023-01-11T22:24:25.1457645Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2023-01-11T22:24:25.1458005Z               )
2023-01-11T22:24:25.1458313Z               (linear1): Linear(in_features=16, out_features=8, bias=True)
2023-01-11T22:24:25.1458648Z               (dropout): Dropout(p=0.1, inplace=False)
2023-01-11T22:24:25.1459004Z               (linear2): Linear(in_features=8, out_features=16, bias=True)
2023-01-11T22:24:25.1459464Z               (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2023-01-11T22:24:25.1459905Z               (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2023-01-11T22:24:25.1460256Z               (dropout1): Dropout(p=0.1, inplace=False)
2023-01-11T22:24:25.1460592Z               (dropout2): Dropout(p=0.1, inplace=False)
2023-01-11T22:24:25.1460864Z             )
2023-01-11T22:24:25.1461067Z           )
2023-01-11T22:24:25.1461283Z         )
2023-01-11T22:24:25.1461653Z         (norm): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2023-01-11T22:24:25.1461928Z       )
2023-01-11T22:24:25.1462185Z       (decoder): TransformerDecoder(
2023-01-11T22:24:25.1462538Z         (layers): ModuleList(
2023-01-11T22:24:25.1462893Z           (0-1): 2 x FullyShardedDataParallel(
2023-01-11T22:24:25.1463258Z             (_fsdp_wrapped_module): TransformerDecoderLayer(
2023-01-11T22:24:25.1463658Z               (self_attn): MultiheadAttention(
2023-01-11T22:24:25.1464058Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2023-01-11T22:24:25.1464416Z               )
2023-01-11T22:24:25.1464698Z               (multihead_attn): MultiheadAttention(
2023-01-11T22:24:25.1465101Z                 (out_proj): NonDynamicallyQuantizableLinear(in_features=16, out_features=16, bias=True)
2023-01-11T22:24:25.1465453Z               )
2023-01-11T22:24:25.1465755Z               (linear1): Linear(in_features=16, out_features=8, bias=True)
2023-01-11T22:24:25.1466104Z               (dropout): Dropout(p=0.1, inplace=False)
2023-01-11T22:24:25.1466442Z               (linear2): Linear(in_features=8, out_features=16, bias=True)
2023-01-11T22:24:25.1466911Z               (norm1): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2023-01-11T22:24:25.1467368Z               (norm2): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2023-01-11T22:24:25.1467798Z               (norm3): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2023-01-11T22:24:25.1468149Z               (dropout1): Dropout(p=0.1, inplace=False)
2023-01-11T22:24:25.1468482Z               (dropout2): Dropout(p=0.1, inplace=False)
2023-01-11T22:24:25.1468794Z               (dropout3): Dropout(p=0.1, inplace=False)
2023-01-11T22:24:25.1469062Z             )
2023-01-11T22:24:25.1469281Z           )
2023-01-11T22:24:25.1469478Z         )
2023-01-11T22:24:25.1469851Z         (norm): LayerNorm((16,), eps=1e-05, elementwise_affine=True)
2023-01-11T22:24:25.1470140Z       )
2023-01-11T22:24:25.1470333Z     )
2023-01-11T22:24:25.1470631Z     (output_proj): Linear(in_features=16, out_features=23, bias=True)
2023-01-11T22:24:25.1471129Z     (bn): BatchNorm1d(2, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
2023-01-11T22:24:25.1471445Z   )
2023-01-11T22:24:25.1471633Z )
2023-01-11T22:24:25.1472013Z ERROR: expected to be in states [<TrainingState.IDLE: 1>] but current state is TrainingState.SUMMON_FULL_PARAMS
2023-01-11T22:24:25.1472389Z ok (4.949s)
2023-01-11T22:24:25.1472653Z   test_nested_module_apply (__main__.TestApply)
2023-01-11T22:24:25.1473274Z Tests that ``apply()`` modifies parameter values in-place on a ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58600
2023-01-11T22:24:25.1473813Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58601
2023-01-11T22:24:25.1474406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:25.1474857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:25.1475433Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:25.1475906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:25.1476470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:25.1476921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:25.1477488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:25.1477949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:25.1478385Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:24:25.1478878Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:24:25.1479537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:25.1480271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:25.1480807Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:25.1481334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:25.1482613Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:24:25.1483390Z   warnings.warn(
2023-01-11T22:24:25.1484718Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:24:25.1485513Z   warnings.warn(
2023-01-11T22:24:25.1485761Z dist init r=1, world=2
2023-01-11T22:24:25.1486008Z dist init r=0, world=2
2023-01-11T22:24:25.1486225Z ok (3.410s)
2023-01-11T22:24:25.1486514Z   test_transformer_module_apply (__main__.TestApply)
2023-01-11T22:24:25.1487154Z Tests that ``apply()`` modifies parameter values in-place on an ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58679
2023-01-11T22:24:25.1487677Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58680
2023-01-11T22:24:25.1488290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:25.1488741Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:25.1489317Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:25.1489774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:25.1490354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:25.1490797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:25.1491354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:25.1491824Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:25.1492278Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:24:25.1492785Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:24:25.1493426Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:25.1494122Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:24:25.1494646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:25.1495116Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:25.1496465Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:24:25.1497256Z   warnings.warn(
2023-01-11T22:24:25.1498485Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:24:25.1499271Z   warnings.warn(
2023-01-11T22:24:25.1499518Z dist init r=1, world=2
2023-01-11T22:24:25.1499751Z dist init r=0, world=2
2023-01-11T22:24:25.1499992Z ok (3.810s)
2023-01-11T22:24:25.1500138Z 
2023-01-11T22:24:25.1500407Z ----------------------------------------------------------------------
2023-01-11T22:24:25.1500730Z Ran 3 tests in 12.170s
2023-01-11T22:24:25.1500890Z 
2023-01-11T22:24:25.1500982Z OK
2023-01-11T22:24:25.1501112Z 
2023-01-11T22:24:25.1501235Z Generating XML reports...
2023-01-11T22:24:25.1501790Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20230111222412.xml
2023-01-11T22:24:25.1502120Z 
2023-01-11T22:24:25.1502449Z ##[endgroup]
2023-01-11T22:24:25.1503042Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_apply (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_apply_u5cypieg)
2023-01-11T22:24:25.1503396Z 
2023-01-11T22:24:25.1503667Z Running distributed/fsdp/test_fsdp_overlap ... [2023-01-11 22:24:25.142723]
2023-01-11T22:24:25.1504331Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_overlap.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:24:25.142979]
2023-01-11T22:24:40.8013279Z 
2023-01-11T22:24:40.8013920Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_overlap
2023-01-11T22:24:40.8015281Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_overlap_8a5sk3_v)
2023-01-11T22:24:40.8015669Z 
2023-01-11T22:24:40.8015765Z Running tests...
2023-01-11T22:24:40.8016526Z ----------------------------------------------------------------------
2023-01-11T22:24:40.8017194Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap
2023-01-11T22:24:40.8018311Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeOne) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:24:40.8019271Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58793
2023-01-11T22:24:40.8020238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:40.8020708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:40.8021298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:40.8021754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:40.8022216Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:24:40.8022879Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:24:40.8023405Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:40.8024173Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2023-01-11T22:24:40.8024655Z   warnings.warn(
2023-01-11T22:24:40.8026083Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:24:40.8026976Z   warnings.warn(
2023-01-11T22:24:40.8027204Z dist init r=0, world=1
2023-01-11T22:24:40.8027367Z 
2023-01-11T22:24:40.8027460Z rank0:
2023-01-11T22:24:40.8027957Z   e1: {'cpu_iter': 0.001797766700000114, 'cpu_wait': 3.554320000000999e-05, 'gpu_compute': 0.0639103996567428, 'gpu_total': 0.7304672002792358}
2023-01-11T22:24:40.8028540Z   e2: {'cpu_iter': 0.005498549500000216, 'cpu_wait': 3.416119999997136e-05, 'gpu_compute': 0.24540479965507983, 'gpu_total': 2.286636781692505}
2023-01-11T22:24:40.8029105Z   e3: {'cpu_iter': 0.0018930447000001572, 'cpu_wait': 0.18597652679999985, 'gpu_compute': 188.23400077819824, 'gpu_total': 188.51773071289062}
2023-01-11T22:24:40.8029671Z   e4: {'cpu_iter': 0.005598841899999663, 'cpu_wait': 0.18320072859999997, 'gpu_compute': 188.2496150970459, 'gpu_total': 188.77910461425782}
2023-01-11T22:24:40.8029996Z ok (13.282s)
2023-01-11T22:24:40.8031042Z   test_forward_overlap (__main__.TestForwardOverlapWorldSizeTwo) ... skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/71183 for allplatform(s) . If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (0.001s)
2023-01-11T22:24:40.8031664Z 
2023-01-11T22:24:40.8031930Z ----------------------------------------------------------------------
2023-01-11T22:24:40.8032242Z Ran 2 tests in 13.283s
2023-01-11T22:24:40.8032405Z 
2023-01-11T22:24:40.8032511Z OK (skipped=1)
2023-01-11T22:24:40.8032661Z 
2023-01-11T22:24:40.8032784Z Generating XML reports...
2023-01-11T22:24:40.8033443Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20230111222427.xml
2023-01-11T22:24:40.8034315Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20230111222427.xml
2023-01-11T22:24:40.8034702Z 
2023-01-11T22:24:40.8035027Z ##[endgroup]
2023-01-11T22:24:40.8035619Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_overlap (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_overlap_8a5sk3_v)
2023-01-11T22:24:40.8035981Z 
2023-01-11T22:24:40.8036237Z Running distributed/_tensor/test_api ... [2023-01-11 22:24:40.801398]
2023-01-11T22:24:40.8036893Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_api.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:24:40.801677]
2023-01-11T22:24:56.6563903Z 
2023-01-11T22:24:56.6564760Z Expand the folded group to see the log file of distributed/_tensor/test_api
2023-01-11T22:24:56.6565935Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_api_kuii4btw)
2023-01-11T22:24:56.6566287Z 
2023-01-11T22:24:56.6566394Z Running tests...
2023-01-11T22:24:56.6566918Z ----------------------------------------------------------------------
2023-01-11T22:24:56.6567474Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_api
2023-01-11T22:24:56.6568300Z   test_distribute_module (__main__.DTensorAPITest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:24:56.6568762Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 58870
2023-01-11T22:24:56.6569220Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 58871
2023-01-11T22:24:56.6569669Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 58872
2023-01-11T22:24:56.6570097Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 58873
2023-01-11T22:24:56.6571001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6571485Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6572188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6572646Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6573235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6573691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6574252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6574748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6575346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6575839Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6576410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6576883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6577469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6577922Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6578559Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6579027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6579474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:24:56.6579941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:56.6580408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:56.6580873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:24:56.6581272Z skip: Need at least 4 CUDA devices (4.009s)
2023-01-11T22:24:56.6581824Z   test_distribute_module_input_fn_output_fn (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59006
2023-01-11T22:24:56.6582352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59007
2023-01-11T22:24:56.6582800Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59008
2023-01-11T22:24:56.6583246Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59009
2023-01-11T22:24:56.6583865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6584301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6584882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6585357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6585921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6586365Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6586946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6587411Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6588048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6601726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6602431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6603060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6603652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6604107Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6605200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6605685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6606129Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:56.6606595Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:24:56.6607075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:56.6607550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:24:56.6607944Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:24:56.6608394Z   test_distribute_tensor (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59142
2023-01-11T22:24:56.6608914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59143
2023-01-11T22:24:56.6609363Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59144
2023-01-11T22:24:56.6609791Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59145
2023-01-11T22:24:56.6610416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6610880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6611461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6611926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6612508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6612952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6613527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6613976Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6614555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6615006Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6615557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6616027Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6616605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6617048Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6617593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6618057Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6618495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:24:56.6619017Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:56.6619630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:56.6620108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:24:56.6620571Z skip: Need at least 4 CUDA devices (2.309s)
2023-01-11T22:24:56.6621052Z   test_distribute_tensor_errors (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59278
2023-01-11T22:24:56.6621562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59279
2023-01-11T22:24:56.6622021Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59280
2023-01-11T22:24:56.6622468Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59281
2023-01-11T22:24:56.6623098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6623542Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6624122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6624602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6625192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6625623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6626194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6626660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6627220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6627662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6628236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6628701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6629267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6629720Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6630292Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6630735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6631177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:56.6631654Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:56.6632130Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:24:56.6632588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:24:56.6632983Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:24:56.6633470Z   test_distribute_tensor_uneven_sharding (__main__.DTensorAPITest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59414
2023-01-11T22:24:56.6634012Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59415
2023-01-11T22:24:56.6634450Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59416
2023-01-11T22:24:56.6634897Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59417
2023-01-11T22:24:56.6635514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6635952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6636601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6637089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6637736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6638169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6638750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6639222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6639787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6640236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6640817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6641288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6641849Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:24:56.6642302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:24:56.6642883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:24:56.6643352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:24:56.6643875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:24:56.6644875Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:24:56.6645413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:24:56.6645877Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:24:56.6646333Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:24:56.6646534Z 
2023-01-11T22:24:56.6646819Z ----------------------------------------------------------------------
2023-01-11T22:24:56.6647156Z Ran 5 tests in 13.550s
2023-01-11T22:24:56.6647302Z 
2023-01-11T22:24:56.6647414Z OK (skipped=5)
2023-01-11T22:24:56.6647569Z 
2023-01-11T22:24:56.6647698Z Generating XML reports...
2023-01-11T22:24:56.6648272Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_api/TEST-DTensorAPITest-20230111222442.xml
2023-01-11T22:24:56.6648603Z 
2023-01-11T22:24:56.6648933Z ##[endgroup]
2023-01-11T22:24:56.6649499Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_api (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_api_kuii4btw)
2023-01-11T22:24:56.6649837Z 
2023-01-11T22:24:56.6650147Z Running distributed/tensor/parallel/test_parallelize_api ... [2023-01-11 22:24:56.656345]
2023-01-11T22:24:56.6650864Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/tensor/parallel/test_parallelize_api.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:24:56.656637]
2023-01-11T22:25:14.8339707Z 
2023-01-11T22:25:14.8340702Z Expand the folded group to see the log file of distributed/tensor/parallel/test_parallelize_api
2023-01-11T22:25:14.8342573Z ##[group]PRINTING LOG FILE of distributed/tensor/parallel/test_parallelize_api (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_parallelize_api_h0q2ktc8)
2023-01-11T22:25:14.8343313Z 
2023-01-11T22:25:14.8343495Z Running tests...
2023-01-11T22:25:14.8344289Z ----------------------------------------------------------------------
2023-01-11T22:25:14.8345553Z Test results will be stored in test-reports/python-unittest/distributed.tensor.parallel.test_parallelize_api
2023-01-11T22:25:14.8346407Z   test_creat_1d_device_mesh (__main__.TensorParallelAPITests) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:25:14.8346922Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59585
2023-01-11T22:25:14.8347469Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59586
2023-01-11T22:25:14.8347912Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59587
2023-01-11T22:25:14.8348362Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59588
2023-01-11T22:25:14.8348990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8349453Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8350039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8350518Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8351175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8351632Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8352196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8352671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8353252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8353702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8354262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8354735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8355323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8355750Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8356324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8356794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8357236Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:14.8357690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:25:14.8358157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:25:14.8358631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:14.8359009Z skip: Need at least 4 CUDA devices (4.025s)
2023-01-11T22:25:14.8359517Z   test_creat_1d_device_mesh_error (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59721
2023-01-11T22:25:14.8360067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59722
2023-01-11T22:25:14.8360523Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59723
2023-01-11T22:25:14.8360952Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59724
2023-01-11T22:25:14.8361563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8362021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8362582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8363054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8363704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8364162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8365023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8365593Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8366178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8366625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8367178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8367648Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8368228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8368662Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8369237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8369705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8370149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:14.8370612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:25:14.8371077Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:14.8371549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:25:14.8371928Z skip: Need at least 4 CUDA devices (2.409s)
2023-01-11T22:25:14.8372429Z   test_linear_col_wise_parallel (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59857
2023-01-11T22:25:14.8372981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59858
2023-01-11T22:25:14.8373437Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59859
2023-01-11T22:25:14.8373866Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59860
2023-01-11T22:25:14.8374476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8374932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8375519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8375977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8376565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8377019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8377577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8378053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8378631Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8379075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8379630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8380104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8380691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8381189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8381777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8382298Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8382745Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:25:14.8383203Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:25:14.8383671Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:14.8384144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:14.8384543Z skip: Need at least 4 CUDA devices (2.310s)
2023-01-11T22:25:14.8385027Z   test_linear_row_wise_parallel (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 59993
2023-01-11T22:25:14.8385585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 59994
2023-01-11T22:25:14.8386039Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 59995
2023-01-11T22:25:14.8386479Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 59996
2023-01-11T22:25:14.8387119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8387574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8388152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8388609Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8389193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8389651Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8390196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8390654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8391229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8391702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8392274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8392745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8393325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8393777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8394340Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8394809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8395253Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:25:14.8395716Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:14.8396184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:25:14.8396657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:14.8397052Z skip: Need at least 4 CUDA devices (2.309s)
2023-01-11T22:25:14.8397521Z   test_parallelize_mlp (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60129
2023-01-11T22:25:14.8398124Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60130
2023-01-11T22:25:14.8398582Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60131
2023-01-11T22:25:14.8399010Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60132
2023-01-11T22:25:14.8399672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8400128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8400705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8401156Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8401738Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8402188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8402750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8403219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8403802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8404432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8404999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8405465Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8406044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8406492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8407052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8407522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8407972Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:25:14.8408432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:25:14.8408904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:14.8409366Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:14.8409762Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:25:14.8410239Z   test_parallelize_mlp_error (__main__.TensorParallelAPITests) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60265
2023-01-11T22:25:14.8410792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60266
2023-01-11T22:25:14.8411250Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 60267
2023-01-11T22:25:14.8411682Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 60268
2023-01-11T22:25:14.8412302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8412752Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8413325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8413780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8414359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8414850Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8415518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8415981Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8416566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8417091Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8417667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8418116Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8418695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:14.8419143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:14.8419742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:14.8420217Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:14.8420659Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:25:14.8421139Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:25:14.8421591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:14.8422052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:14.8422448Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:25:14.8422640Z 
2023-01-11T22:25:14.8422916Z ----------------------------------------------------------------------
2023-01-11T22:25:14.8423232Z Ran 6 tests in 15.874s
2023-01-11T22:25:14.8423396Z 
2023-01-11T22:25:14.8423504Z OK (skipped=6)
2023-01-11T22:25:14.8423662Z 
2023-01-11T22:25:14.8423787Z Generating XML reports...
2023-01-11T22:25:14.8424440Z Generated XML report: test-reports/python-unittest/distributed.tensor.parallel.test_parallelize_api/TEST-TensorParallelAPITests-20230111222458.xml
2023-01-11T22:25:14.8424848Z 
2023-01-11T22:25:14.8425185Z ##[endgroup]
2023-01-11T22:25:14.8425869Z FINISHED PRINTING LOG FILE of distributed/tensor/parallel/test_parallelize_api (/var/lib/jenkins/workspace/test/test-reports/distributed-tensor-parallel-test_parallelize_api_h0q2ktc8)
2023-01-11T22:25:14.8426287Z 
2023-01-11T22:25:14.8426551Z Running distributed/fsdp/test_fsdp_hybrid_shard ... [2023-01-11 22:25:14.834125]
2023-01-11T22:25:14.8427249Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_hybrid_shard.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:25:14.834421]
2023-01-11T22:25:33.6283511Z 
2023-01-11T22:25:33.6284470Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_hybrid_shard
2023-01-11T22:25:33.6285733Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_hybrid_shard (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_hybrid_shard_ifvfpi9w)
2023-01-11T22:25:33.6286123Z 
2023-01-11T22:25:33.6286245Z Running tests...
2023-01-11T22:25:33.6286858Z ----------------------------------------------------------------------
2023-01-11T22:25:33.6287449Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_hybrid_shard
2023-01-11T22:25:33.6287911Z   test_fsdp_hybrid_shard_basic_setup (__main__.TestFSDPHybridShard)
2023-01-11T22:25:33.6288397Z Tests basic functionality of HYBRID_SHARD and _HYBRID_SHARD_ZERO2: ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:25:33.6288907Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60436
2023-01-11T22:25:33.6289346Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60437
2023-01-11T22:25:33.6289986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:33.6290696Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:33.6291308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:33.6291871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:33.6292464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:33.6292917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:33.6293533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:33.6294039Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:33.6294501Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:25:33.6295005Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:25:33.6295658Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:33.6296359Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:33.6296891Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:33.6297373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:33.6297847Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:25:33.6298342Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:25:33.6299004Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:25:33.6299523Z INFO:torch.distributed.distributed_c10d:Rank 0 is assigned to subgroup [0, 1]
2023-01-11T22:25:33.6299989Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:25:33.6300652Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:25:33.6301164Z INFO:torch.distributed.distributed_c10d:Rank 1 is assigned to subgroup [0, 1]
2023-01-11T22:25:33.6301624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:25:33.6302277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:25:33.6302811Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T22:25:33.6303463Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:25:33.6303982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T22:25:33.6304628Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2023-01-11T22:25:33.6305312Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2023-01-11T22:25:33.6305846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T22:25:33.6306321Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T22:25:33.6306976Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes.
2023-01-11T22:25:33.6307492Z INFO:torch.distributed.distributed_c10d:Rank 1 is assigned to subgroup [0, 1]
2023-01-11T22:25:33.6308040Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 1
2023-01-11T22:25:33.6308689Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes.
2023-01-11T22:25:33.6309259Z INFO:torch.distributed.distributed_c10d:Rank 0 is assigned to subgroup [0, 1]
2023-01-11T22:25:33.6309737Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:6 to store for rank: 0
2023-01-11T22:25:33.6310380Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes.
2023-01-11T22:25:33.6310916Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 0
2023-01-11T22:25:33.6311566Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:6 with 2 nodes.
2023-01-11T22:25:33.6312096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:7 to store for rank: 1
2023-01-11T22:25:33.6312733Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes.
2023-01-11T22:25:33.6313416Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:7 with 2 nodes.
2023-01-11T22:25:33.6313954Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 1
2023-01-11T22:25:33.6314606Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes.
2023-01-11T22:25:33.6315126Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:8 to store for rank: 0
2023-01-11T22:25:33.6315595Z INFO:torch.distributed.distributed_c10d:Rank 1 is assigned to subgroup [0, 1]
2023-01-11T22:25:33.6316226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:8 with 2 nodes.
2023-01-11T22:25:33.6316745Z INFO:torch.distributed.distributed_c10d:Rank 0 is assigned to subgroup [0, 1]
2023-01-11T22:25:33.6317205Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 1
2023-01-11T22:25:33.6317699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:9 to store for rank: 0
2023-01-11T22:25:33.6318355Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes.
2023-01-11T22:25:33.6318876Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 0
2023-01-11T22:25:33.6319531Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:9 with 2 nodes.
2023-01-11T22:25:33.6320069Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:10 to store for rank: 1
2023-01-11T22:25:33.6320780Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes.
2023-01-11T22:25:33.6321445Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:10 with 2 nodes.
2023-01-11T22:25:33.6321988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 1
2023-01-11T22:25:33.6322490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:11 to store for rank: 0
2023-01-11T22:25:33.6323148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes.
2023-01-11T22:25:33.6323636Z INFO:torch.distributed.distributed_c10d:Rank 0 is assigned to subgroup [0, 1]
2023-01-11T22:25:33.6324114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 0
2023-01-11T22:25:33.6324982Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:11 with 2 nodes.
2023-01-11T22:25:33.6325593Z INFO:torch.distributed.distributed_c10d:Rank 1 is assigned to subgroup [0, 1]
2023-01-11T22:25:33.6326064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:12 to store for rank: 1
2023-01-11T22:25:33.6326786Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes.
2023-01-11T22:25:33.6327317Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 1
2023-01-11T22:25:33.6327954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:12 with 2 nodes.
2023-01-11T22:25:33.6328486Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:13 to store for rank: 0
2023-01-11T22:25:33.6329145Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes.
2023-01-11T22:25:33.6329836Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:13 with 2 nodes.
2023-01-11T22:25:33.6330879Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6332152Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6333411Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6334664Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6335910Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6337138Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6338378Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6339610Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6340904Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6342141Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6343422Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6344733Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6345979Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6347218Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6348454Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6349685Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6350920Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6352150Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6353386Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6354596Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6355920Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6357161Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6358450Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6359689Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6360441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 1
2023-01-11T22:25:33.6361095Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes.
2023-01-11T22:25:33.6361639Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:14 to store for rank: 0
2023-01-11T22:25:33.6362124Z INFO:torch.distributed.distributed_c10d:Rank 1 is assigned to subgroup [0, 1]
2023-01-11T22:25:33.6362762Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:14 with 2 nodes.
2023-01-11T22:25:33.6363257Z INFO:torch.distributed.distributed_c10d:Rank 0 is assigned to subgroup [0, 1]
2023-01-11T22:25:33.6363739Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 1
2023-01-11T22:25:33.6364527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:15 to store for rank: 0
2023-01-11T22:25:33.6365290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes.
2023-01-11T22:25:33.6365832Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 0
2023-01-11T22:25:33.6366495Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:15 with 2 nodes.
2023-01-11T22:25:33.6367020Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:16 to store for rank: 1
2023-01-11T22:25:33.6367681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes.
2023-01-11T22:25:33.6368368Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:16 with 2 nodes.
2023-01-11T22:25:33.6368906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 1
2023-01-11T22:25:33.6369389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:17 to store for rank: 0
2023-01-11T22:25:33.6370046Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes.
2023-01-11T22:25:33.6370562Z INFO:torch.distributed.distributed_c10d:Rank 0 is assigned to subgroup [0, 1]
2023-01-11T22:25:33.6371043Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 0
2023-01-11T22:25:33.6371680Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:17 with 2 nodes.
2023-01-11T22:25:33.6372189Z INFO:torch.distributed.distributed_c10d:Rank 1 is assigned to subgroup [0, 1]
2023-01-11T22:25:33.6372671Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:18 to store for rank: 1
2023-01-11T22:25:33.6373395Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes.
2023-01-11T22:25:33.6373941Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 1
2023-01-11T22:25:33.6374701Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:18 with 2 nodes.
2023-01-11T22:25:33.6375239Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:19 to store for rank: 0
2023-01-11T22:25:33.6375875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes.
2023-01-11T22:25:33.6376558Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:19 with 2 nodes.
2023-01-11T22:25:33.6377140Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 1
2023-01-11T22:25:33.6377796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes.
2023-01-11T22:25:33.6378312Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:20 to store for rank: 0
2023-01-11T22:25:33.6378791Z INFO:torch.distributed.distributed_c10d:Rank 1 is assigned to subgroup [0, 1]
2023-01-11T22:25:33.6379425Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:20 with 2 nodes.
2023-01-11T22:25:33.6379933Z INFO:torch.distributed.distributed_c10d:Rank 0 is assigned to subgroup [0, 1]
2023-01-11T22:25:33.6380396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 1
2023-01-11T22:25:33.6380894Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:21 to store for rank: 0
2023-01-11T22:25:33.6381560Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes.
2023-01-11T22:25:33.6382082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 0
2023-01-11T22:25:33.6382739Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:21 with 2 nodes.
2023-01-11T22:25:33.6383276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:22 to store for rank: 1
2023-01-11T22:25:33.6383928Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes.
2023-01-11T22:25:33.6384594Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:22 with 2 nodes.
2023-01-11T22:25:33.6385130Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 1
2023-01-11T22:25:33.6385626Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:23 to store for rank: 0
2023-01-11T22:25:33.6386282Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes.
2023-01-11T22:25:33.6386774Z INFO:torch.distributed.distributed_c10d:Rank 0 is assigned to subgroup [0, 1]
2023-01-11T22:25:33.6387253Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 0
2023-01-11T22:25:33.6387903Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:23 with 2 nodes.
2023-01-11T22:25:33.6388408Z INFO:torch.distributed.distributed_c10d:Rank 1 is assigned to subgroup [0, 1]
2023-01-11T22:25:33.6388868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:24 to store for rank: 1
2023-01-11T22:25:33.6389513Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes.
2023-01-11T22:25:33.6390049Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 1
2023-01-11T22:25:33.6390889Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:24 with 2 nodes.
2023-01-11T22:25:33.6391694Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:25 to store for rank: 0
2023-01-11T22:25:33.6392458Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes.
2023-01-11T22:25:33.6393141Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:25 with 2 nodes.
2023-01-11T22:25:33.6394179Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6395513Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6396769Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6398017Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6399242Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6400487Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6401712Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6402953Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6404365Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6405617Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6406948Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6408200Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6409500Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6410737Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6411967Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6413206Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6414438Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6415668Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6416890Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6418121Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6419346Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6420618Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6421909Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6423229Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6423888Z dist init r=0, world=2
2023-01-11T22:25:33.6424145Z 0 created process group for [0]
2023-01-11T22:25:33.6424433Z 0 created process group for [0]
2023-01-11T22:25:33.6424721Z 0 created process group for [0]
2023-01-11T22:25:33.6424985Z 0 created process group for [0]
2023-01-11T22:25:33.6425267Z 0 created process group for [0]
2023-01-11T22:25:33.6425549Z 0 created process group for [0]
2023-01-11T22:25:33.6425812Z 0 created process group for [0]
2023-01-11T22:25:33.6426091Z 0 created process group for [0]
2023-01-11T22:25:33.6426363Z dist init r=1, world=2
2023-01-11T22:25:33.6426614Z 1 created process group for [1]
2023-01-11T22:25:33.6426895Z 1 created process group for [1]
2023-01-11T22:25:33.6427178Z 1 created process group for [1]
2023-01-11T22:25:33.6427438Z 1 created process group for [1]
2023-01-11T22:25:33.6427754Z 1 created process group for [1]
2023-01-11T22:25:33.6428035Z 1 created process group for [1]
2023-01-11T22:25:33.6428296Z 1 created process group for [1]
2023-01-11T22:25:33.6428580Z 1 created process group for [1]
2023-01-11T22:25:33.6428833Z ok (6.534s)
2023-01-11T22:25:33.6429276Z   test_hybrid_shard_pg_mismatch_raises (__main__.TestFSDPHybridShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60611
2023-01-11T22:25:33.6429830Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60612
2023-01-11T22:25:33.6430455Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:33.6430923Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:33.6431496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:33.6431980Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:33.6432568Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:33.6433019Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:33.6433578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:33.6434054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:33.6434517Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:25:33.6435006Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:25:33.6435663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:33.6436362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:33.6436890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:33.6437350Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:33.6437830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 1
2023-01-11T22:25:33.6438328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:2 to store for rank: 0
2023-01-11T22:25:33.6438984Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:25:33.6439718Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:2 with 2 nodes.
2023-01-11T22:25:33.6440264Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 1
2023-01-11T22:25:33.6440806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:3 to store for rank: 0
2023-01-11T22:25:33.6441463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:25:33.6442131Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:3 with 2 nodes.
2023-01-11T22:25:33.6442722Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 1
2023-01-11T22:25:33.6443215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:4 to store for rank: 0
2023-01-11T22:25:33.6443853Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2023-01-11T22:25:33.6444801Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:4 with 2 nodes.
2023-01-11T22:25:33.6445348Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 1
2023-01-11T22:25:33.6445846Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:5 to store for rank: 0
2023-01-11T22:25:33.6446484Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes.
2023-01-11T22:25:33.6447163Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:5 with 2 nodes.
2023-01-11T22:25:33.6448216Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6449470Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:25:33.6450091Z dist init r=0, world=2
2023-01-11T22:25:33.6450329Z dist init r=1, world=2
2023-01-11T22:25:33.6450569Z ok (3.310s)
2023-01-11T22:25:33.6451032Z   test_invalid_pg_specification_raises (__main__.TestFSDPHybridShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60698
2023-01-11T22:25:33.6451569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60699
2023-01-11T22:25:33.6452197Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:33.6452654Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:33.6453235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:33.6453695Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:33.6454270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:33.6454721Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:33.6455295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:33.6455745Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:33.6456201Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:25:33.6456784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:25:33.6457443Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:33.6458203Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:33.6458730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:33.6459211Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:33.6459552Z dist init r=0, world=2
2023-01-11T22:25:33.6459807Z dist init r=1, world=2
2023-01-11T22:25:33.6460050Z ok (3.309s)
2023-01-11T22:25:33.6460505Z   test_raises_manual_wrap_hybrid_shard_when_none_policy (__main__.TestFSDPHybridShard) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60777
2023-01-11T22:25:33.6461076Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60778
2023-01-11T22:25:33.6461689Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:33.6462145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:33.6462705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:33.6463178Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:33.6463760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:33.6464192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:33.6464766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:33.6465235Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:33.6465692Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:25:33.6466179Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:25:33.6466844Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:33.6467539Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:33.6468064Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:33.6468523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:33.6468970Z dist init r=1, world=2
2023-01-11T22:25:33.6469225Z dist init r=0, world=2
2023-01-11T22:25:33.6469447Z ok (3.309s)
2023-01-11T22:25:33.6469597Z 
2023-01-11T22:25:33.6469872Z ----------------------------------------------------------------------
2023-01-11T22:25:33.6470204Z Ran 4 tests in 16.462s
2023-01-11T22:25:33.6470367Z 
2023-01-11T22:25:33.6470447Z OK
2023-01-11T22:25:33.6470582Z 
2023-01-11T22:25:33.6470705Z Generating XML reports...
2023-01-11T22:25:33.6471324Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_hybrid_shard/TEST-TestFSDPHybridShard-20230111222516.xml
2023-01-11T22:25:33.6471692Z 
2023-01-11T22:25:33.6472018Z ##[endgroup]
2023-01-11T22:25:33.6472644Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_hybrid_shard (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_hybrid_shard_ifvfpi9w)
2023-01-11T22:25:33.6473014Z 
2023-01-11T22:25:33.6473323Z Running distributed/checkpoint/test_file_system_checkpoint ... [2023-01-11 22:25:33.628745]
2023-01-11T22:25:33.6474133Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:25:33.629028]
2023-01-11T22:25:55.6017144Z 
2023-01-11T22:25:55.6019941Z Expand the folded group to see the log file of distributed/checkpoint/test_file_system_checkpoint
2023-01-11T22:25:55.6021265Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_j2iu_0nk)
2023-01-11T22:25:55.6021768Z 
2023-01-11T22:25:55.6023560Z Running tests...
2023-01-11T22:25:55.6024234Z ----------------------------------------------------------------------
2023-01-11T22:25:55.6024862Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint
2023-01-11T22:25:55.6025446Z   test_load_rowwise_to_colwise (__main__.TestDistributedReshardOnLoad) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:25:55.6025946Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60891
2023-01-11T22:25:55.6026423Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60892
2023-01-11T22:25:55.6030410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:55.6030926Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:55.6031536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:55.6032025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:55.6032605Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:55.6033067Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:55.6033662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:55.6034148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:55.6034583Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:55.6035075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:55.6035585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:25:55.6036176Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:25:55.6036831Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:55.6037535Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:55.6037940Z ok (4.952s)
2023-01-11T22:25:55.6038470Z   test_load_with_different_shard_plan (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 60974
2023-01-11T22:25:55.6039030Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 60975
2023-01-11T22:25:55.6039653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:55.6040118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:55.6040705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:55.6041163Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:55.6041749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:55.6042203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:55.6042785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:55.6043427Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:55.6043897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:55.6044900Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:25:55.6045381Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:55.6045875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:25:55.6046558Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:55.6047258Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:55.6047641Z ok (3.912s)
2023-01-11T22:25:55.6048105Z   test_save_load_bytes (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61057
2023-01-11T22:25:55.6048659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61058
2023-01-11T22:25:55.6049283Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:55.6049724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:55.6050305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:55.6050780Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:55.6051345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:55.6051794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:55.6052377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:55.6052848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:55.6053279Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:55.6053760Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:55.6054261Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:25:55.6054745Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:25:55.6055417Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:55.6056113Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:55.6056511Z ok (3.309s)
2023-01-11T22:25:55.6056989Z   test_switch_between_sharded_tensor_to_tensor (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61140
2023-01-11T22:25:55.6057569Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61141
2023-01-11T22:25:55.6058218Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:55.6058693Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:55.6059261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:55.6059737Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:55.6060327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:55.6060755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:55.6061430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:55.6061921Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:55.6062439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:55.6062921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:25:55.6063412Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:55.6063906Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:25:55.6064579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:55.6065261Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:55.6065662Z ok (4.111s)
2023-01-11T22:25:55.6066855Z   test_read_write_only_tensor (__main__.TestDistributedStateDictSaveLoad) ... /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:25:55.6067814Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:25:55.6068103Z ok (0.047s)
2023-01-11T22:25:55.6068615Z   test_read_write_shard_tensor (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61223
2023-01-11T22:25:55.6069239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61224
2023-01-11T22:25:55.6069866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:55.6070306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:55.6070896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:55.6071370Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:55.6071958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:25:55.6072390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:25:55.6072968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:25:55.6073438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:25:55.6073869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:25:55.6074367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:25:55.6074864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:25:55.6075362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:25:55.6076008Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:55.6076708Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:25:55.6077961Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:25:55.6078733Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:25:55.6079050Z ok (3.309s)
2023-01-11T22:25:55.6079205Z 
2023-01-11T22:25:55.6079485Z ----------------------------------------------------------------------
2023-01-11T22:25:55.6079818Z Ran 6 tests in 19.642s
2023-01-11T22:25:55.6079984Z 
2023-01-11T22:25:55.6080080Z OK
2023-01-11T22:25:55.6080198Z 
2023-01-11T22:25:55.6080324Z Generating XML reports...
2023-01-11T22:25:55.6081015Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20230111222535.xml
2023-01-11T22:25:55.6081962Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20230111222535.xml
2023-01-11T22:25:55.6082996Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20230111222535.xml
2023-01-11T22:25:55.6083498Z 
2023-01-11T22:25:55.6083889Z ##[endgroup]
2023-01-11T22:25:55.6084992Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_j2iu_0nk)
2023-01-11T22:25:55.6085414Z 
2023-01-11T22:25:55.6085692Z Running distributed/test_c10d_spawn_ucc ... [2023-01-11 22:25:55.601918]
2023-01-11T22:25:55.6086386Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_ucc.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:25:55.602204]
2023-01-11T22:26:41.4036952Z 
2023-01-11T22:26:41.4037665Z Expand the folded group to see the log file of distributed/test_c10d_spawn_ucc
2023-01-11T22:26:41.4038613Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_ucc (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_ucc_enzthkm8)
2023-01-11T22:26:41.4042483Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0axdyjee
2023-01-11T22:26:41.4043082Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0axdyjee/_remote_module_non_scriptable.py
2023-01-11T22:26:41.4043544Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:41.4043857Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:26:41.4044174Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:26:41.4046071Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsUcc testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsUcc testMethod=test_reduce>]>
2023-01-11T22:26:41.4047223Z test_all_gather (__main__.TestDistributedNNFunctionsUcc)
2023-01-11T22:26:41.4047625Z test_all_to_all (__main__.TestDistributedNNFunctionsUcc)
2023-01-11T22:26:41.4048050Z test_all_to_all_single (__main__.TestDistributedNNFunctionsUcc)
2023-01-11T22:26:41.4048466Z test_allreduce (__main__.TestDistributedNNFunctionsUcc)
2023-01-11T22:26:41.4048866Z test_broadcast (__main__.TestDistributedNNFunctionsUcc)
2023-01-11T22:26:41.4049240Z test_reduce (__main__.TestDistributedNNFunctionsUcc)
2023-01-11T22:26:41.4049962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:41.4050430Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:41.4051021Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:41.4051507Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:41.4052255Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpockoigst
2023-01-11T22:26:41.4052839Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpockoigst/_remote_module_non_scriptable.py
2023-01-11T22:26:41.4053509Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:41.4053842Z 
2023-01-11T22:26:41.4053959Z Running tests...
2023-01-11T22:26:41.4054382Z ----------------------------------------------------------------------
2023-01-11T22:26:41.4054914Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2023-01-11T22:26:41.4055503Z   test_all_gather (__main__.TestDistributedNNFunctionsUcc) ... skip: runs into illegal memory access on first assertEqual check when run locally (0.000s)
2023-01-11T22:26:41.4055870Z 
2023-01-11T22:26:41.4056142Z ----------------------------------------------------------------------
2023-01-11T22:26:41.4056481Z Ran 1 test in 0.001s
2023-01-11T22:26:41.4056646Z 
2023-01-11T22:26:41.4056738Z OK (skipped=1)
2023-01-11T22:26:41.4056895Z 
2023-01-11T22:26:41.4057024Z Generating XML reports...
2023-01-11T22:26:41.4057697Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111222601.xml
2023-01-11T22:26:41.4058458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:41.4058908Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:41.4059501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:41.4059989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:41.4060493Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoee3akoo
2023-01-11T22:26:41.4061032Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoee3akoo/_remote_module_non_scriptable.py
2023-01-11T22:26:41.4061464Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:41.4061662Z 
2023-01-11T22:26:41.4061778Z Running tests...
2023-01-11T22:26:41.4062194Z ----------------------------------------------------------------------
2023-01-11T22:26:41.4062722Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2023-01-11T22:26:41.4063313Z   test_all_to_all (__main__.TestDistributedNNFunctionsUcc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61409
2023-01-11T22:26:41.4063857Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61410
2023-01-11T22:26:41.4064456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:41.4064920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:41.4065506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:41.4065984Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:41.4066558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:41.4067014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:41.4067602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:41.4068072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:41.4068533Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdcuzwjys
2023-01-11T22:26:41.4069086Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdcuzwjys/_remote_module_non_scriptable.py
2023-01-11T22:26:41.4069639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpusar8wgi
2023-01-11T22:26:41.4070163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpusar8wgi/_remote_module_non_scriptable.py
2023-01-11T22:26:41.4070666Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:41.4071088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:41.4071635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:26:41.4072028Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:41.4072437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:41.4072930Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:26:41.4073591Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:41.4074293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:41.4074878Z [1673475968.667646] [c3943a31ca1f:61409:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2023-01-11T22:26:41.4075401Z [1673475968.679341] [c3943a31ca1f:61409:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2023-01-11T22:26:41.4075869Z [1673475968.679341] [c3943a31ca1f:61409:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2023-01-11T22:26:41.4076390Z [1673475968.676956] [c3943a31ca1f:61410:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2023-01-11T22:26:41.4076895Z [1673475968.686608] [c3943a31ca1f:61410:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2023-01-11T22:26:41.4077366Z [1673475968.686608] [c3943a31ca1f:61410:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2023-01-11T22:26:41.4077692Z ok (4.425s)
2023-01-11T22:26:41.4077843Z 
2023-01-11T22:26:41.4078124Z ----------------------------------------------------------------------
2023-01-11T22:26:41.4078453Z Ran 1 test in 4.426s
2023-01-11T22:26:41.4078617Z 
2023-01-11T22:26:41.4078712Z OK
2023-01-11T22:26:41.4078935Z 
2023-01-11T22:26:41.4079067Z Generating XML reports...
2023-01-11T22:26:41.4079714Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111222605.xml
2023-01-11T22:26:41.4080447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:41.4080902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:41.4081487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:41.4081943Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:41.4082414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpipzleadn
2023-01-11T22:26:41.4082964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpipzleadn/_remote_module_non_scriptable.py
2023-01-11T22:26:41.4083404Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:41.4083588Z 
2023-01-11T22:26:41.4083697Z Running tests...
2023-01-11T22:26:41.4084111Z ----------------------------------------------------------------------
2023-01-11T22:26:41.4087654Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2023-01-11T22:26:41.4088231Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsUcc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61530
2023-01-11T22:26:41.4088787Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61531
2023-01-11T22:26:41.4089406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:41.4089863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:41.4090541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:41.4091033Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:41.4091698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:41.4092147Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:41.4092704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:41.4093174Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:41.4093649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpck3wa6jy
2023-01-11T22:26:41.4094180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpck3wa6jy/_remote_module_non_scriptable.py
2023-01-11T22:26:41.4094724Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptazan4ow
2023-01-11T22:26:41.4095262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptazan4ow/_remote_module_non_scriptable.py
2023-01-11T22:26:41.4095700Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:41.4096008Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:41.4096414Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:41.4096912Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:26:41.4097388Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:41.4097880Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:26:41.4098553Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:41.4099260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:41.4099819Z [1673475976.397187] [c3943a31ca1f:61530:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2023-01-11T22:26:41.4100338Z [1673475976.409707] [c3943a31ca1f:61530:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2023-01-11T22:26:41.4100822Z [1673475976.409707] [c3943a31ca1f:61530:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2023-01-11T22:26:41.4101341Z [1673475976.402688] [c3943a31ca1f:61531:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2023-01-11T22:26:41.4101829Z [1673475976.413098] [c3943a31ca1f:61531:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2023-01-11T22:26:41.4102307Z [1673475976.413098] [c3943a31ca1f:61531:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2023-01-11T22:26:41.4102651Z ok (4.424s)
2023-01-11T22:26:41.4102803Z 
2023-01-11T22:26:41.4103077Z ----------------------------------------------------------------------
2023-01-11T22:26:41.4103386Z Ran 1 test in 4.425s
2023-01-11T22:26:41.4103547Z 
2023-01-11T22:26:41.4103641Z OK
2023-01-11T22:26:41.4103774Z 
2023-01-11T22:26:41.4103899Z Generating XML reports...
2023-01-11T22:26:41.4104529Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111222612.xml
2023-01-11T22:26:41.4105274Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:41.4105734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:41.4106381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:41.4106850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:41.4107326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp88lt8vdo
2023-01-11T22:26:41.4107926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp88lt8vdo/_remote_module_non_scriptable.py
2023-01-11T22:26:41.4108341Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:41.4108543Z 
2023-01-11T22:26:41.4108651Z Running tests...
2023-01-11T22:26:41.4109061Z ----------------------------------------------------------------------
2023-01-11T22:26:41.4109602Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2023-01-11T22:26:41.4110160Z   test_allreduce (__main__.TestDistributedNNFunctionsUcc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61651
2023-01-11T22:26:41.4110709Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61652
2023-01-11T22:26:41.4111333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:41.4111785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:41.4112353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:41.4112827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:41.4113412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:41.4113848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:41.4114428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:41.4114900Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:41.4115375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpah20l7v2
2023-01-11T22:26:41.4115901Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpah20l7v2/_remote_module_non_scriptable.py
2023-01-11T22:26:41.4116447Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdcn9u39n
2023-01-11T22:26:41.4116988Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdcn9u39n/_remote_module_non_scriptable.py
2023-01-11T22:26:41.4117403Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:41.4117810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:41.4118302Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:26:41.4118709Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:41.4119099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:41.4119593Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:26:41.4120265Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:41.4120951Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:41.4121530Z [1673475984.097861] [c3943a31ca1f:61651:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2023-01-11T22:26:41.4122042Z [1673475984.110383] [c3943a31ca1f:61651:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2023-01-11T22:26:41.4122529Z [1673475984.110383] [c3943a31ca1f:61651:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2023-01-11T22:26:41.4123119Z [1673475984.102772] [c3943a31ca1f:61652:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2023-01-11T22:26:41.4123698Z [1673475984.114317] [c3943a31ca1f:61652:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2023-01-11T22:26:41.4124597Z [1673475984.114317] [c3943a31ca1f:61652:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2023-01-11T22:26:41.4124956Z ok (4.526s)
2023-01-11T22:26:41.4125107Z 
2023-01-11T22:26:41.4125372Z ----------------------------------------------------------------------
2023-01-11T22:26:41.4125701Z Ran 1 test in 4.526s
2023-01-11T22:26:41.4125864Z 
2023-01-11T22:26:41.4125956Z OK
2023-01-11T22:26:41.4126090Z 
2023-01-11T22:26:41.4126215Z Generating XML reports...
2023-01-11T22:26:41.4126849Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111222620.xml
2023-01-11T22:26:41.4127613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:41.4128071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:41.4128636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:41.4129118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:41.4129588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4s716674
2023-01-11T22:26:41.4130133Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4s716674/_remote_module_non_scriptable.py
2023-01-11T22:26:41.4130546Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:41.4130744Z 
2023-01-11T22:26:41.4130851Z Running tests...
2023-01-11T22:26:41.4131259Z ----------------------------------------------------------------------
2023-01-11T22:26:41.4131782Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2023-01-11T22:26:41.4132365Z   test_broadcast (__main__.TestDistributedNNFunctionsUcc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61770
2023-01-11T22:26:41.4132910Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61771
2023-01-11T22:26:41.4133528Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:41.4133967Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:41.4134546Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:41.4135022Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:41.4135608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:41.4136037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:41.4136617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:41.4137089Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:41.4137545Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0n_gajrn
2023-01-11T22:26:41.4138088Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0n_gajrn/_remote_module_non_scriptable.py
2023-01-11T22:26:41.4138624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjumf5w6i
2023-01-11T22:26:41.4139168Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjumf5w6i/_remote_module_non_scriptable.py
2023-01-11T22:26:41.4139580Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:41.4139991Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:41.4140487Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:26:41.4140962Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:41.4141384Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:41.4141875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:26:41.4142611Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:41.4143292Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:41.4143865Z [1673475991.942964] [c3943a31ca1f:61770:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2023-01-11T22:26:41.4144382Z [1673475991.956025] [c3943a31ca1f:61770:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2023-01-11T22:26:41.4144867Z [1673475991.956025] [c3943a31ca1f:61770:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2023-01-11T22:26:41.4145367Z [1673475991.945548] [c3943a31ca1f:61771:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2023-01-11T22:26:41.4145873Z [1673475991.955914] [c3943a31ca1f:61771:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2023-01-11T22:26:41.4146346Z [1673475991.955914] [c3943a31ca1f:61771:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2023-01-11T22:26:41.4146690Z ok (4.527s)
2023-01-11T22:26:41.4146823Z 
2023-01-11T22:26:41.4147097Z ----------------------------------------------------------------------
2023-01-11T22:26:41.4147428Z Ran 1 test in 4.527s
2023-01-11T22:26:41.4147590Z 
2023-01-11T22:26:41.4147684Z OK
2023-01-11T22:26:41.4147818Z 
2023-01-11T22:26:41.4147926Z Generating XML reports...
2023-01-11T22:26:41.4148577Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111222628.xml
2023-01-11T22:26:41.4149324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:41.4149788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:41.4150350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:41.4150821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:41.4151294Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgq84admm
2023-01-11T22:26:41.4151825Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgq84admm/_remote_module_non_scriptable.py
2023-01-11T22:26:41.4152256Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:41.4152456Z 
2023-01-11T22:26:41.4152565Z Running tests...
2023-01-11T22:26:41.4152980Z ----------------------------------------------------------------------
2023-01-11T22:26:41.4153509Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_ucc
2023-01-11T22:26:41.4154088Z   test_reduce (__main__.TestDistributedNNFunctionsUcc) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 61889
2023-01-11T22:26:41.4154632Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 61890
2023-01-11T22:26:41.4155231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:41.4155686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:41.4156256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:26:41.4156713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:26:41.4157335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:41.4157822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:41.4158472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:26:41.4158945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:26:41.4159402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxtw686p2
2023-01-11T22:26:41.4159948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxtw686p2/_remote_module_non_scriptable.py
2023-01-11T22:26:41.4160484Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgkjll21y
2023-01-11T22:26:41.4161005Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgkjll21y/_remote_module_non_scriptable.py
2023-01-11T22:26:41.4161436Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:41.4161855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:26:41.4162351Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:26:41.4162743Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:26:41.4163147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:26:41.4163637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:26:41.4164680Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:41.4165403Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:26:41.4165981Z [1673475999.737726] [c3943a31ca1f:61890:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2023-01-11T22:26:41.4166501Z [1673475999.747535] [c3943a31ca1f:61890:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2023-01-11T22:26:41.4166962Z [1673475999.747535] [c3943a31ca1f:61890:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2023-01-11T22:26:41.4167571Z [1673475999.730521] [c3943a31ca1f:61889:0]         ec_cuda.c:343  cuda ec WARN  CUDA cooperative groups are not supported. Fall back to non cooperative launch.
2023-01-11T22:26:41.4168081Z [1673475999.742924] [c3943a31ca1f:61889:0]          parser.c:1993 UCX  WARN  unused environment variables: UCX_COMMIT; UCX_HOME
2023-01-11T22:26:41.4168553Z [1673475999.742924] [c3943a31ca1f:61889:0]          parser.c:1993 UCX  WARN  (set UCX_WARN_UNUSED_ENV_VARS=n to suppress this warning)
2023-01-11T22:26:41.4168878Z ok (4.425s)
2023-01-11T22:26:41.4169028Z 
2023-01-11T22:26:41.4169302Z ----------------------------------------------------------------------
2023-01-11T22:26:41.4169636Z Ran 1 test in 4.425s
2023-01-11T22:26:41.4169799Z 
2023-01-11T22:26:41.4169892Z OK
2023-01-11T22:26:41.4170008Z 
2023-01-11T22:26:41.4170133Z Generating XML reports...
2023-01-11T22:26:41.4170780Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111222636.xml
2023-01-11T22:26:41.4171169Z 
2023-01-11T22:26:41.4171577Z ##[endgroup]
2023-01-11T22:26:41.4172154Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_ucc (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_ucc_enzthkm8)
2023-01-11T22:26:41.4172497Z 
2023-01-11T22:26:41.4172810Z Running distributed/algorithms/ddp_comm_hooks/test_ddp_hooks ... [2023-01-11 22:26:41.403940]
2023-01-11T22:26:41.4173556Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/ddp_comm_hooks/test_ddp_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:26:41.404189]
2023-01-11T22:27:10.0768330Z 
2023-01-11T22:27:10.0769319Z Expand the folded group to see the log file of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks
2023-01-11T22:27:10.0772875Z ##[group]PRINTING LOG FILE of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-ddp_comm_hooks-test_ddp_hooks_fri7gbnh)
2023-01-11T22:27:10.0773578Z 
2023-01-11T22:27:10.0773681Z Running tests...
2023-01-11T22:27:10.0774437Z ----------------------------------------------------------------------
2023-01-11T22:27:10.0775313Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks
2023-01-11T22:27:10.0775865Z   test_ddp_comm_hook_allreduce_hook (__main__.DistributedDataParallelCommHookTest)
2023-01-11T22:27:10.0776406Z This unit test verifies the ``allreduce`` hook registered case gives same result ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:27:10.0776927Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62008
2023-01-11T22:27:10.0777396Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62009
2023-01-11T22:27:10.0778044Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:10.0778493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:10.0779081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:10.0779564Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:10.0780164Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:10.0780607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:10.0781190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:10.0781673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:10.0782108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:10.0782597Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:10.0783223Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:27:10.0784163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:27:10.0785596Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:27:10.0786457Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:27:10.0787029Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcj2wgmhc
2023-01-11T22:27:10.0787594Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcj2wgmhc/_remote_module_non_scriptable.py
2023-01-11T22:27:10.0788117Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptmp6c1er
2023-01-11T22:27:10.0788676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptmp6c1er/_remote_module_non_scriptable.py
2023-01-11T22:27:10.0789060Z ok (5.356s)
2023-01-11T22:27:10.0789424Z   test_ddp_comm_hook_fp16compress_hook (__main__.DistributedDataParallelCommHookTest)
2023-01-11T22:27:10.0790012Z This unit test verifies the ``fp16 compress`` hook registered case ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62092
2023-01-11T22:27:10.0791291Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62093
2023-01-11T22:27:10.0791937Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:10.0792397Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:10.0793093Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:10.0793594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:10.0794249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:10.0795094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:10.0796241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:10.0797173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:10.0797628Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:10.0798088Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:10.0798592Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:27:10.0799094Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:27:10.0799779Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:27:10.0800467Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:27:10.0801026Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufmliz4e
2023-01-11T22:27:10.0801581Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufmliz4e/_remote_module_non_scriptable.py
2023-01-11T22:27:10.0802123Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsiz1ypj3
2023-01-11T22:27:10.0802650Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsiz1ypj3/_remote_module_non_scriptable.py
2023-01-11T22:27:10.0803043Z ok (3.710s)
2023-01-11T22:27:10.0803415Z   test_ddp_comm_hook_noop_hook (__main__.DistributedDataParallelCommHookTest)
2023-01-11T22:27:10.0803990Z This unit test verifies the ``noop`` hook registered case and a subsequent allreduce ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62176
2023-01-11T22:27:10.0805110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62177
2023-01-11T22:27:10.0805749Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:10.0806210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:10.0806775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:10.0807252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:10.0807844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:10.0808275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:10.0808851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:10.0809327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:10.0809775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:10.0810238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:10.0810731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:27:10.0811235Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:27:10.0812038Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:27:10.0812748Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:27:10.0813378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyzcawe7v
2023-01-11T22:27:10.0813927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyzcawe7v/_remote_module_non_scriptable.py
2023-01-11T22:27:10.0814452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6wu8nhr
2023-01-11T22:27:10.0814995Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6wu8nhr/_remote_module_non_scriptable.py
2023-01-11T22:27:10.0815388Z ok (3.810s)
2023-01-11T22:27:10.0815779Z   test_ddp_comm_hook_quantize_per_channel_hook (__main__.DistributedDataParallelCommHookTest)
2023-01-11T22:27:10.0816359Z This unit test verifies the ``quantize per channel`` hook registered case ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62260
2023-01-11T22:27:10.0816916Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62261
2023-01-11T22:27:10.0817543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:10.0818005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:10.0818558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:10.0819013Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:10.0819591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:10.0820051Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:10.0820648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:10.0821126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:10.0821571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:10.0822035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:10.0822526Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:27:10.0823025Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:27:10.0823672Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:27:10.0824370Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:27:10.0824984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpseyaw0hs
2023-01-11T22:27:10.0825543Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpseyaw0hs/_remote_module_non_scriptable.py
2023-01-11T22:27:10.0826127Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmhj0yl8j
2023-01-11T22:27:10.0826656Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmhj0yl8j/_remote_module_non_scriptable.py
2023-01-11T22:27:10.0827044Z ok (3.810s)
2023-01-11T22:27:10.0827432Z   test_ddp_comm_hook_quantize_per_tensor_hook (__main__.DistributedDataParallelCommHookTest)
2023-01-11T22:27:10.0828012Z This unit test verifies the ``quantize per tensor`` hook registered case ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62344
2023-01-11T22:27:10.0828563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62345
2023-01-11T22:27:10.0829194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:10.0829717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:10.0830297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:10.0830834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:10.0831423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:10.0831874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:10.0832435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:10.0832910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:10.0833354Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:10.0833818Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:10.0834316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:27:10.0834816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:27:10.0835494Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:27:10.0836176Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:27:10.0836735Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo45mj2l8
2023-01-11T22:27:10.0837272Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7cqiqxr
2023-01-11T22:27:10.0837813Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo45mj2l8/_remote_module_non_scriptable.py
2023-01-11T22:27:10.0838351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7cqiqxr/_remote_module_non_scriptable.py
2023-01-11T22:27:10.0838734Z ok (3.810s)
2023-01-11T22:27:10.0839203Z   test_is_last_hook (__main__.DistributedDataParallelCommHookTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62428
2023-01-11T22:27:10.0839751Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62429
2023-01-11T22:27:10.0840368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:10.0840828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:10.0841411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:10.0841865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:10.0842454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:10.0842909Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:10.0843466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:10.0843940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:10.0844947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:10.0845444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:10.0845920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:27:10.0846418Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:27:10.0847101Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:27:10.0847899Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:27:10.0848455Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp138sapg9
2023-01-11T22:27:10.0849085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp138sapg9/_remote_module_non_scriptable.py
2023-01-11T22:27:10.0849626Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy1sfl0jp
2023-01-11T22:27:10.0850149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy1sfl0jp/_remote_module_non_scriptable.py
2023-01-11T22:27:10.0850538Z ok (5.914s)
2023-01-11T22:27:10.0850691Z 
2023-01-11T22:27:10.0850967Z ----------------------------------------------------------------------
2023-01-11T22:27:10.0851303Z Ran 6 tests in 26.412s
2023-01-11T22:27:10.0851450Z 
2023-01-11T22:27:10.0851545Z OK
2023-01-11T22:27:10.0851681Z 
2023-01-11T22:27:10.0851808Z Generating XML reports...
2023-01-11T22:27:10.0852541Z Generated XML report: test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20230111222643.xml
2023-01-11T22:27:10.0852991Z 
2023-01-11T22:27:10.0853318Z ##[endgroup]
2023-01-11T22:27:10.0854020Z FINISHED PRINTING LOG FILE of distributed/algorithms/ddp_comm_hooks/test_ddp_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-ddp_comm_hooks-test_ddp_hooks_fri7gbnh)
2023-01-11T22:27:10.0854436Z 
2023-01-11T22:27:10.0854716Z Running distributed/_tensor/test_common_rules ... [2023-01-11 22:27:10.076956]
2023-01-11T22:27:10.0855407Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_common_rules.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:27:10.077222]
2023-01-11T22:27:40.1785219Z 
2023-01-11T22:27:40.1787920Z Expand the folded group to see the log file of distributed/_tensor/test_common_rules
2023-01-11T22:27:40.1788905Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_common_rules (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_common_rules_0exs79tu)
2023-01-11T22:27:40.1789280Z 
2023-01-11T22:27:40.1789480Z Running tests...
2023-01-11T22:27:40.1790147Z ----------------------------------------------------------------------
2023-01-11T22:27:40.1791681Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_common_rules
2023-01-11T22:27:40.1792455Z   test_einop_basic_propagation (__main__.CommonRulesTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:27:40.1793262Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62547
2023-01-11T22:27:40.1793722Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62548
2023-01-11T22:27:40.1794177Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62549
2023-01-11T22:27:40.1796152Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62550
2023-01-11T22:27:40.1796888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1797371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1797955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1798393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1798962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1801039Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1801904Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1802521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1803377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1803849Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1804869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1805488Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1806578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1807408Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1808625Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1809404Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1810378Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:40.1811050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:27:40.1811527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:27:40.1811988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:40.1812378Z skip: Need at least 4 CUDA devices (4.008s)
2023-01-11T22:27:40.1812844Z   test_einop_errors (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62683
2023-01-11T22:27:40.1813363Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62684
2023-01-11T22:27:40.1813799Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62685
2023-01-11T22:27:40.1814236Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62686
2023-01-11T22:27:40.1814863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1815309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1815892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1816367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1816956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1817386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1817962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1818431Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1818993Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1819447Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1820028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1820496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1821061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1821512Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1822088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1822557Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1822976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:40.1823450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:27:40.1824060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:27:40.1824529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:40.1824974Z skip: Need at least 4 CUDA devices (2.311s)
2023-01-11T22:27:40.1825443Z   test_einop_linearity (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62819
2023-01-11T22:27:40.1825966Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62820
2023-01-11T22:27:40.1826456Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62821
2023-01-11T22:27:40.1826896Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62822
2023-01-11T22:27:40.1827518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1827955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1828617Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1829092Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1829669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1830097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1830672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1831170Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1831729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1832174Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1832746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1833215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1833781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1834235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1834808Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1835274Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1835697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:40.1836177Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:27:40.1836651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:27:40.1837104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:40.1837491Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:27:40.1837967Z   test_einop_merge_sharding (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 62955
2023-01-11T22:27:40.1838495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 62956
2023-01-11T22:27:40.1838927Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 62957
2023-01-11T22:27:40.1839367Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 62958
2023-01-11T22:27:40.1839983Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1840418Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1841060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1841537Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1842175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1842825Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1843411Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1843881Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1844783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1845237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1845821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1846290Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1846850Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1847305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1847873Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1848334Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1848753Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:27:40.1849229Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:40.1849702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:27:40.1850160Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:40.1850557Z skip: Need at least 4 CUDA devices (2.310s)
2023-01-11T22:27:40.1851050Z   test_einop_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63091
2023-01-11T22:27:40.1851585Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63092
2023-01-11T22:27:40.1852019Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63093
2023-01-11T22:27:40.1852460Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63094
2023-01-11T22:27:40.1853078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1853516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1854097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1854568Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1855152Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1855584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1856160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1856628Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1857210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1857637Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1858210Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1858773Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1859353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1859895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1860472Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1860940Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1861361Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:27:40.1861841Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:27:40.1862316Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:40.1862775Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:40.1863171Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:27:40.1863657Z   test_einop_pointwise_propagation (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63227
2023-01-11T22:27:40.1864197Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63228
2023-01-11T22:27:40.1864626Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63229
2023-01-11T22:27:40.1865063Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63230
2023-01-11T22:27:40.1865677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1866133Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1866691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1867165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1867746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1868181Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1868755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1869218Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1869797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1870226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1870805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1871273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1871836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1872285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1872858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1873327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1873748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:27:40.1874228Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:27:40.1874701Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:40.1875155Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:40.1875607Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:27:40.1876133Z   test_pointwise_enforce_sharding_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63363
2023-01-11T22:27:40.1876748Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63364
2023-01-11T22:27:40.1877185Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63365
2023-01-11T22:27:40.1877622Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63366
2023-01-11T22:27:40.1878234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1878684Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1879241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1879714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1880300Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1880729Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1881304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1881771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1882348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1882774Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1883353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1883818Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1884802Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1885237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1885824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1886289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1886711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:27:40.1887186Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:27:40.1887662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:40.1888133Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:40.1888510Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:27:40.1889002Z   test_pointwise_multi_sharding_on_mesh_dim (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63499
2023-01-11T22:27:40.1889543Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63500
2023-01-11T22:27:40.1889975Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63501
2023-01-11T22:27:40.1890424Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63502
2023-01-11T22:27:40.1891030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1891478Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1892037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1892506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1893183Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1893623Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1894263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1894735Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1895315Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1895743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1896321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1896831Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1897414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1897866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1898427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1898897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1899338Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:27:40.1899797Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:40.1900269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:27:40.1900731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:40.1901125Z skip: Need at least 4 CUDA devices (2.411s)
2023-01-11T22:27:40.1901596Z   test_pointwise_rules_broadcasting (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63635
2023-01-11T22:27:40.1902132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63636
2023-01-11T22:27:40.1902588Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63637
2023-01-11T22:27:40.1903015Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63638
2023-01-11T22:27:40.1903628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1904074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1904648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1905099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1905680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1906122Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1906694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1907140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1907712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1908154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1908711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1909180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1909833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1910285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1910842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1911369Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1911804Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:40.1912263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:27:40.1912738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:40.1913202Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:27:40.1913596Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:27:40.1914064Z   test_pointwise_rules_suggestion (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63771
2023-01-11T22:27:40.1914595Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63772
2023-01-11T22:27:40.1915051Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63773
2023-01-11T22:27:40.1915500Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63774
2023-01-11T22:27:40.1916098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1916554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1917132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1917587Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1918177Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1918625Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1919200Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1919654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1920233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1935264Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1935953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1936451Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1937073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1937564Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1938180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1938672Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1939145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:27:40.1939652Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:40.1940136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:27:40.1940638Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:40.1941053Z skip: Need at least 4 CUDA devices (2.410s)
2023-01-11T22:27:40.1941534Z   test_reduction_rule (__main__.CommonRulesTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 63907
2023-01-11T22:27:40.1942233Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 63908
2023-01-11T22:27:40.1942739Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 63909
2023-01-11T22:27:40.1943283Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 63910
2023-01-11T22:27:40.1943947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1944435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1945032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1945542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1946167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1946655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1947254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1947764Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1948386Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1948866Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1949461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1949967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1950588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:27:40.1951053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:27:40.1951676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:27:40.1952184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:27:40.1952658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:27:40.1953146Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:27:40.1953651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T22:27:40.1954156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T22:27:40.1954558Z skip: Need at least 4 CUDA devices (2.310s)
2023-01-11T22:27:40.1954766Z 
2023-01-11T22:27:40.1955058Z ----------------------------------------------------------------------
2023-01-11T22:27:40.1955415Z Ran 11 tests in 27.814s
2023-01-11T22:27:40.1955590Z 
2023-01-11T22:27:40.1955713Z OK (skipped=11)
2023-01-11T22:27:40.1955858Z 
2023-01-11T22:27:40.1955990Z Generating XML reports...
2023-01-11T22:27:40.1956622Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_common_rules/TEST-CommonRulesTest-20230111222711.xml
2023-01-11T22:27:40.1956998Z 
2023-01-11T22:27:40.1957492Z ##[endgroup]
2023-01-11T22:27:40.1958128Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_common_rules (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_common_rules_0exs79tu)
2023-01-11T22:27:40.1958518Z 
2023-01-11T22:27:40.1958824Z Running distributed/fsdp/test_fsdp_clip_grad_norm ... [2023-01-11 22:27:40.178805]
2023-01-11T22:27:40.1959580Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_clip_grad_norm.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:27:40.179091]
2023-01-11T22:28:13.0307551Z 
2023-01-11T22:28:13.0308050Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_clip_grad_norm
2023-01-11T22:28:13.0312687Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_clip_grad_norm_tm3xi5e3)
2023-01-11T22:28:13.0313327Z 
2023-01-11T22:28:13.0313532Z Running tests...
2023-01-11T22:28:13.0314377Z ----------------------------------------------------------------------
2023-01-11T22:28:13.0315535Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm
2023-01-11T22:28:13.0316448Z   test_ddp_parity (__main__.TestClipGradNorm)
2023-01-11T22:28:13.0317476Z Tests FSDP with ``FullyShardedDataParallel.clip_grad_norm_()` against ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:28:13.0318531Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64078
2023-01-11T22:28:13.0319085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64079
2023-01-11T22:28:13.0320107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:13.0320864Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:13.0321844Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:13.0322941Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:13.0324130Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:13.0325393Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:13.0326487Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:13.0327352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:13.0328465Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:28:13.0329334Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:28:13.0330621Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:13.0332003Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:13.0333006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:13.0333477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:13.0333960Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0334449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0335481Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0336747Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0337979Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0339370Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0340180Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0340676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0341160Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0341625Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0342631Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0343871Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0345118Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0346353Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0347587Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0348802Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0350036Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0351270Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0352003Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0352491Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0353480Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0354763Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0356050Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0357283Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0358515Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0359735Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0360963Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0362193Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0363432Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0364970Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0366203Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0367436Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0368657Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0369962Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0371251Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0372480Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0373711Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0374943Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0376168Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0377396Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0378137Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0378633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0379624Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0380860Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0382066Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0383308Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0384046Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0384812Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0385896Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0387198Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0387911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0388400Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0389403Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0390633Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0391871Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0393085Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0393829Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0394319Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0395558Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0396810Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0398042Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0399264Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0400563Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0401802Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0403088Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0404681Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0405927Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0407152Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0407888Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0408362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0409361Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0410587Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0411821Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0413052Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0413789Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0414261Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0415256Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0416577Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0417890Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0419125Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0420349Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0421566Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0422786Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0424019Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0425248Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0426471Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0428005Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0429249Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0430472Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0431770Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0432540Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0433034Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0434034Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0435257Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0435986Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0436456Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0437697Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:795: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:28:13.0438534Z   return torch._VF.split_with_sizes(self, split_size, dim)
2023-01-11T22:28:13.0439705Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:795: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:28:13.0440519Z   return torch._VF.split_with_sizes(self, split_size, dim)
2023-01-11T22:28:13.0440931Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0441418Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0442425Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0443676Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0445277Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0446505Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0447824Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0449138Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0450376Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0451609Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0452346Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0452838Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0453303Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0453790Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0454789Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0456027Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0457264Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0458480Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0459712Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0460946Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0462222Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0463465Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0464242Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0464732Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0465710Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0466946Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0468178Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0469407Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0470628Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0471854Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0473068Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0474281Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0475507Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0476736Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0477524Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0478024Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0479020Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0480304Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0481031Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0481523Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0482517Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0483240Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0483702Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0485045Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0486286Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0487528Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0488254Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0488719Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0489718Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0491684Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0492948Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0494292Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0495040Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0495582Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0496587Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0497813Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0499046Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0500268Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0501005Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0501476Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0502483Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0503720Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0504954Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0506186Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0507422Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0508648Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0509915Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0511472Z /opt/conda/lib/python3.10/site-packages/torch/nn/functional.py:4772: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:28:13.0512394Z   return (linear(q, w_q, b_q),) + linear(k, w_kv, b_kv).chunk(2, dim=-1)
2023-01-11T22:28:13.0512837Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0513330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0514319Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0515574Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0516816Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0518048Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0519281Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0520511Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0521743Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0522974Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0524616Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0525889Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0527218Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0528498Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0529733Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0530955Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0532192Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0533411Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0534642Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0535849Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0537079Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0538304Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0539573Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0540815Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0542101Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0543336Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0544564Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0545790Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0546998Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0548235Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0549466Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0550688Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0551918Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0553149Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0554380Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0555661Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0557031Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0558248Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0559464Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0560695Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0561922Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0563135Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0564678Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0565925Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0567157Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0568384Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0569613Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0570922Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0572216Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0573424Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0574654Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0575393Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0575887Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0576878Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0578111Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0579336Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0580070Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0580557Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0581554Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0582789Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0584011Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0585286Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0586518Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0587807Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0589046Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0590262Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0590997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0591486Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0591951Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0592433Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0593431Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0594674Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0595889Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0597122Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0597861Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0598349Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0599341Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0600620Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0601854Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0603131Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0604661Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0605912Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0607147Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0608434Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0609965Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0611254Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0612489Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0613698Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0614439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0614930Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0615923Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0617255Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0618558Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0619782Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0621023Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0622262Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0623489Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0624224Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0624714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0625693Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0626917Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0627648Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0628189Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0629184Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0630402Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0631689Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0632935Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0634229Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0634970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0635458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0636438Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0637672Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0638402Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0638886Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0639881Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0641124Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0642334Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0643571Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0645126Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0646365Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0647102Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0647681Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0648673Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0649985Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0651223Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0653043Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0654373Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0655620Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0656853Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0658074Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0659293Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0660533Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0661772Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0663087Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0664356Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0665636Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0666890Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0667647Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0668142Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0669119Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0670346Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0671577Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0672307Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0672794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:13.0673780Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0674389Z dist init r=0, world=2
2023-01-11T22:28:13.0674642Z dist init r=1, world=2
2023-01-11T22:28:13.0674883Z ok (22.480s)
2023-01-11T22:28:13.0675169Z   test_low_precision_grads (__main__.TestClipGradNorm)
2023-01-11T22:28:13.0675678Z Tests ``clip_grad_norm_()`` when using low precision gradients. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64161
2023-01-11T22:28:13.0676213Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64162
2023-01-11T22:28:13.0676826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:13.0677294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:13.0677890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:13.0678374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:13.0678957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:13.0679488Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:13.0680083Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:13.0680607Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:13.0681045Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:28:13.0681544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:28:13.0682208Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:13.0682910Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:13.0683419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:13.0683898Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:13.0685261Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0686522Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0687778Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0689049Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0690289Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0691516Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0692752Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0693980Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0695309Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0696575Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0697897Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0699126Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0700665Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0701942Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0703174Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0704865Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0706920Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0708193Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0709429Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0710661Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0711985Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0713235Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0714528Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0715767Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0716986Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0718224Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0719451Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0720675Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0721906Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0723143Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0724796Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0726066Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0727384Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0728695Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0730020Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0731260Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0732481Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0733715Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0734939Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0736170Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0737398Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0738618Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0739849Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0741091Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0742371Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:28:13.0742989Z dist init r=1, world=2
2023-01-11T22:28:13.0743243Z dist init r=0, world=2
2023-01-11T22:28:13.0743466Z ok (4.311s)
2023-01-11T22:28:13.0743818Z   test_non_root (__main__.TestClipGradNorm)
2023-01-11T22:28:13.0744459Z Tests that calling ``clip_grad_norm_()`` on a non-root FSDP instance ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64244
2023-01-11T22:28:13.0744990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64245
2023-01-11T22:28:13.0745608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:13.0746064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:13.0746645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:13.0747108Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:13.0747693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:13.0748152Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:13.0748716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:13.0749186Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:13.0749645Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:28:13.0750147Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:28:13.0750797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:13.0751501Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:13.0752034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:13.0752515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:13.0752856Z dist init r=0, world=2
2023-01-11T22:28:13.0753111Z dist init r=1, world=2
2023-01-11T22:28:13.0753352Z ok (3.710s)
2023-01-11T22:28:13.0753486Z 
2023-01-11T22:28:13.0753759Z ----------------------------------------------------------------------
2023-01-11T22:28:13.0754090Z Ran 3 tests in 30.502s
2023-01-11T22:28:13.0754253Z 
2023-01-11T22:28:13.0754347Z OK
2023-01-11T22:28:13.0754482Z 
2023-01-11T22:28:13.0754590Z Generating XML reports...
2023-01-11T22:28:13.0755210Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20230111222742.xml
2023-01-11T22:28:13.0755573Z 
2023-01-11T22:28:13.0755945Z ##[endgroup]
2023-01-11T22:28:13.0756568Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_clip_grad_norm (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_clip_grad_norm_tm3xi5e3)
2023-01-11T22:28:13.0756951Z 
2023-01-11T22:28:13.0757232Z Running distributed/_composable/test_compose ... [2023-01-11 22:28:13.031542]
2023-01-11T22:28:13.0757930Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_composable/test_compose.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:28:13.031847]
2023-01-11T22:28:47.3750584Z 
2023-01-11T22:28:47.3751267Z Expand the folded group to see the log file of distributed/_composable/test_compose
2023-01-11T22:28:47.3752238Z ##[group]PRINTING LOG FILE of distributed/_composable/test_compose (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_compose__m56w21a)
2023-01-11T22:28:47.3756011Z 
2023-01-11T22:28:47.3758390Z Running tests...
2023-01-11T22:28:47.3759188Z ----------------------------------------------------------------------
2023-01-11T22:28:47.3760446Z Test results will be stored in test-reports/python-unittest/distributed._composable.test_compose
2023-01-11T22:28:47.3761343Z   test_checkpoint_fsdp_submodules_non_reentrant (__main__.TestFSDPCheckpoint) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:28:47.3762305Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64362
2023-01-11T22:28:47.3762980Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64363
2023-01-11T22:28:47.3763973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:47.3765437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:47.3766531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:47.3768127Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:47.3768756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:47.3769199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:47.3769803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:47.3770284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:47.3771118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:28:47.3771609Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:28:47.3772290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:47.3773024Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:47.3773566Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:47.3774036Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:47.3774409Z dist init r=0, world=2
2023-01-11T22:28:47.3774671Z dist init r=1, world=2
2023-01-11T22:28:47.3774921Z ok (5.528s)
2023-01-11T22:28:47.3775384Z   test_checkpoint_fsdp_submodules_use_reentrant (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64445
2023-01-11T22:28:47.3775953Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64446
2023-01-11T22:28:47.3776584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:47.3777030Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:47.3777624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:47.3778107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:47.3778705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:47.3779150Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:47.3779736Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:47.3780215Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:47.3780706Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:28:47.3781192Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:28:47.3781864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:47.3782717Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:47.3783334Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:47.3783800Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:47.3784160Z dist init r=0, world=2
2023-01-11T22:28:47.3784418Z dist init r=1, world=2
2023-01-11T22:28:47.3784643Z ok (3.711s)
2023-01-11T22:28:47.3785112Z   test_checkpoint_fsdp_submodules_with_param (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64528
2023-01-11T22:28:47.3785670Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64529
2023-01-11T22:28:47.3786301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:47.3786747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:47.3787333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:47.3787815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:47.3788383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:47.3788837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:47.3789424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:47.3789895Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:47.3790341Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:28:47.3790849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:28:47.3791594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:47.3792303Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:47.3792833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:47.3793294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:47.3793659Z dist init r=0, world=2
2023-01-11T22:28:47.3793921Z dist init r=1, world=2
2023-01-11T22:28:47.3794166Z ok (3.810s)
2023-01-11T22:28:47.3794621Z   test_checkpoint_fsdp_submodules_with_param_no_shard (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64611
2023-01-11T22:28:47.3795192Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64612
2023-01-11T22:28:47.3795820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:47.3796260Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:47.3796852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:47.3797326Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:47.3797915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:47.3798346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:47.3798930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:47.3799405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:47.3799913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:28:47.3800427Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:28:47.3801148Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:47.3801855Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:47.3802367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:47.3802846Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:47.3803202Z dist init r=0, world=2
2023-01-11T22:28:47.3803459Z dist init r=1, world=2
2023-01-11T22:28:47.3803683Z ok (3.811s)
2023-01-11T22:28:47.3804140Z   test_composable_fsdp_replicate (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64694
2023-01-11T22:28:47.3805052Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64695
2023-01-11T22:28:47.3805676Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:47.3806129Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:47.3806715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:47.3807200Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:47.3807772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:47.3808225Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:47.3808810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:47.3809266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:47.3809727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:28:47.3810233Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:28:47.3810898Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:47.3811578Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:47.3812108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:47.3812588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:47.3813879Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:28:47.3814754Z   warnings.warn(
2023-01-11T22:28:47.3815916Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:28:47.3816702Z   warnings.warn(
2023-01-11T22:28:47.3816961Z dist init r=0, world=2
2023-01-11T22:28:47.3817316Z dist init r=1, world=2
2023-01-11T22:28:47.3817551Z ok (3.309s)
2023-01-11T22:28:47.3817892Z   test_fully_shard_replicate_composability (__main__.TestFSDPCheckpoint)
2023-01-11T22:28:47.3818505Z Tests composing ``fully_shard`` and ``replicate``. To save unit test ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64773
2023-01-11T22:28:47.3819047Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64774
2023-01-11T22:28:47.3819662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:47.3820119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:47.3820705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:47.3821165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:47.3821752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:47.3822206Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:47.3822794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:47.3823249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:47.3823712Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:28:47.3824214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:28:47.3824866Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:47.3825569Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:47.3826106Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:47.3826588Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:47.3827058Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3827542Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3828020Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3828495Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3828947Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3829480Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3829957Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3830402Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3830880Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3831364Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3831839Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3832288Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3832772Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3833248Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3833716Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3834224Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3834712Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3835230Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3835679Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3836152Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3836623Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3837091Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3837538Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3838016Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3838491Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3838945Z INFO:torch.distributed._composable._ddp:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:28:47.3839299Z dist init r=0, world=2
2023-01-11T22:28:47.3839555Z dist init r=1, world=2
2023-01-11T22:28:47.3839799Z ok (4.211s)
2023-01-11T22:28:47.3840249Z   test_wrap_same_submodule_use_reentrant_False (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64856
2023-01-11T22:28:47.3840814Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64857
2023-01-11T22:28:47.3841442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:47.3841886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:47.3842478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:47.3842958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:47.3843554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:47.3843985Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:47.3844887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:47.3845362Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:47.3845801Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:28:47.3846303Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:28:47.3846981Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:47.3847684Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:47.3848199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:47.3848679Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:47.3849039Z dist init r=0, world=2
2023-01-11T22:28:47.3849276Z dist init r=1, world=2
2023-01-11T22:28:47.3849529Z ok (3.810s)
2023-01-11T22:28:47.3849994Z   test_wrap_same_submodule_use_reentrant_True (__main__.TestFSDPCheckpoint) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 64939
2023-01-11T22:28:47.3850550Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 64940
2023-01-11T22:28:47.3851261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:47.3851730Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:47.3852379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:47.3852857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:47.3853425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:28:47.3853878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:28:47.3854459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:28:47.3854914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:28:47.3855377Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:28:47.3855884Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:28:47.3856553Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:47.3857241Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:28:47.3857769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:28:47.3858249Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:28:47.3858592Z dist init r=1, world=2
2023-01-11T22:28:47.3858850Z dist init r=0, world=2
2023-01-11T22:28:47.3859094Z ok (3.810s)
2023-01-11T22:28:47.3859247Z 
2023-01-11T22:28:47.3859522Z ----------------------------------------------------------------------
2023-01-11T22:28:47.3859836Z Ran 8 tests in 32.002s
2023-01-11T22:28:47.3860003Z 
2023-01-11T22:28:47.3860098Z OK
2023-01-11T22:28:47.3860234Z 
2023-01-11T22:28:47.3860362Z Generating XML reports...
2023-01-11T22:28:47.3860961Z Generated XML report: test-reports/python-unittest/distributed._composable.test_compose/TEST-TestFSDPCheckpoint-20230111222814.xml
2023-01-11T22:28:47.3861329Z 
2023-01-11T22:28:47.3861759Z ##[endgroup]
2023-01-11T22:28:47.3862379Z FINISHED PRINTING LOG FILE of distributed/_composable/test_compose (/var/lib/jenkins/workspace/test/test-reports/distributed-_composable-test_compose__m56w21a)
2023-01-11T22:28:47.3862749Z 
2023-01-11T22:28:47.3863069Z Running distributed/checkpoint/test_file_system_checkpoint_cpu ... [2023-01-11 22:28:47.375402]
2023-01-11T22:28:47.3863815Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/checkpoint/test_file_system_checkpoint_cpu.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:28:47.375780]
2023-01-11T22:29:23.0159828Z 
2023-01-11T22:29:23.0160404Z Expand the folded group to see the log file of distributed/checkpoint/test_file_system_checkpoint_cpu
2023-01-11T22:29:23.0164674Z ##[group]PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint_cpu (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_cpu_guxrkltv)
2023-01-11T22:29:23.0165162Z 
2023-01-11T22:29:23.0165280Z Running tests...
2023-01-11T22:29:23.0165790Z ----------------------------------------------------------------------
2023-01-11T22:29:23.0166410Z Test results will be stored in test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu
2023-01-11T22:29:23.0167018Z   test_load_rowwise_to_colwise_thread_count_1 (__main__.TestDistributedReshardOnLoad) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:29:23.0167536Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65057
2023-01-11T22:29:23.0168001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65058
2023-01-11T22:29:23.0168871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0169348Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0170023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0170498Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0171371Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0172221Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0173453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0174016Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0174471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:23.0174936Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:23.0175433Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:23.0175935Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:23.0182777Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0183508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0184748Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:23.0185576Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:29:23.0186653Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:23.0187409Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:29:23.0187694Z ok (4.845s)
2023-01-11T22:29:23.0188160Z   test_load_rowwise_to_colwise_thread_count_2 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65133
2023-01-11T22:29:23.0188740Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65134
2023-01-11T22:29:23.0189364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0189806Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0190394Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0190871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0191454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0191886Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0192466Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0192939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0193512Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:23.0193986Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:23.0194535Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:23.0195035Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:23.0195790Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0196492Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0197680Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:23.0198422Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:29:23.0199483Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:23.0200228Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:29:23.0200504Z ok (3.210s)
2023-01-11T22:29:23.0200975Z   test_load_with_different_shard_plan_thread_count_1 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65211
2023-01-11T22:29:23.0201562Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65212
2023-01-11T22:29:23.0202185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0202645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0203213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0203689Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0204820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0205634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0206735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0207602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0208363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:23.0208833Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:23.0209329Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:23.0209835Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:23.0210508Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0211186Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0212508Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:23.0213280Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:29:23.0214420Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:23.0215167Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:29:23.0215428Z ok (3.411s)
2023-01-11T22:29:23.0215923Z   test_load_with_different_shard_plan_thread_count_2 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65287
2023-01-11T22:29:23.0216515Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65288
2023-01-11T22:29:23.0217113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0217574Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0218154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0218630Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0219193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0219643Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0220223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0220674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0221124Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:23.0221601Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:23.0222096Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:23.0222577Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:23.0223240Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0223932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0225114Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:23.0225853Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:29:23.0226912Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:23.0227652Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:29:23.0227930Z ok (3.511s)
2023-01-11T22:29:23.0228384Z   test_save_load_bytes_thread_count_1 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65387
2023-01-11T22:29:23.0229011Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65388
2023-01-11T22:29:23.0229639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0230149Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0230768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0231246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0231830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0232280Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0232833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0233302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0233748Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:23.0234214Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:23.0234711Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:23.0235210Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:23.0235875Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0236554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0236952Z ok (2.408s)
2023-01-11T22:29:23.0237417Z   test_save_load_bytes_thread_count_2 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65461
2023-01-11T22:29:23.0237981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65462
2023-01-11T22:29:23.0238576Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0239032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0239608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0240063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0240647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0241094Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0241669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0242122Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0242565Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:23.0243047Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:23.0243533Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:23.0244012Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:23.0245251Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0245953Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0246332Z ok (2.408s)
2023-01-11T22:29:23.0246939Z   test_switch_between_sharded_tensor_to_tensor_thread_count_1 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65537
2023-01-11T22:29:23.0247592Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65538
2023-01-11T22:29:23.0248322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0248785Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0249397Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0249901Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0250524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0250987Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0251604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0252106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0252561Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:23.0253066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:23.0253585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:23.0254116Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:23.0254798Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0255534Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0256787Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:23.0257609Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:29:23.0258718Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:23.0259526Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:29:23.0259816Z ok (3.411s)
2023-01-11T22:29:23.0260351Z   test_switch_between_sharded_tensor_to_tensor_thread_count_2 (__main__.TestDistributedReshardOnLoad) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65613
2023-01-11T22:29:23.0260963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65614
2023-01-11T22:29:23.0261620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0262100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0262700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0263162Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0263773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0264275Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0264964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0265453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0265975Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:23.0266482Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:23.0266982Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:23.0267516Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:23.0268221Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0268961Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0270200Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:23.0271015Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:29:23.0272135Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:23.0272939Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:29:23.0273228Z ok (3.511s)
2023-01-11T22:29:23.0274491Z   test_read_write_only_tensor_thread_count_1 (__main__.TestDistributedStateDictSaveLoad) ... /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:23.0275435Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:29:23.0275727Z ok (0.040s)
2023-01-11T22:29:23.0276130Z   test_read_write_only_tensor_thread_count_2 (__main__.TestDistributedStateDictSaveLoad) ... ok (0.014s)
2023-01-11T22:29:23.0276840Z   test_read_write_shard_tensor_thread_count_1 (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65722
2023-01-11T22:29:23.0277516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65723
2023-01-11T22:29:23.0278173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0278655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0279257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0279754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0280377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0280837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0281448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0281949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0282476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:23.0282974Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:23.0283548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:23.0284079Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:23.0285330Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0286017Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0287206Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:23.0287962Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:29:23.0289015Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:23.0289765Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:29:23.0290026Z ok (3.209s)
2023-01-11T22:29:23.0290570Z   test_read_write_shard_tensor_thread_count_2 (__main__.TestDistributedStateDictSaveLoadWithSharedTensor) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65798
2023-01-11T22:29:23.0291217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65799
2023-01-11T22:29:23.0291817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0292276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0292855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0293330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0293898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:29:23.0294349Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:29:23.0294927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:29:23.0295398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:29:23.0295827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:29:23.0296306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:29:23.0296806Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:29:23.0297289Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:29:23.0297954Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0298651Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:29:23.0299942Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:23.0300772Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:29:23.0301820Z /opt/conda/lib/python3.10/site-packages/torch/distributed/checkpoint/filesystem.py:157: UserWarning: TypedStorage is deprecated. It will be removed in the future and UntypedStorage will be the only storage class. This should only matter to you if you are using storages directly.  To access UntypedStorage directly, use tensor.untyped_storage() instead of tensor.storage()
2023-01-11T22:29:23.0302566Z   if tensor.storage().size() != tensor.numel():
2023-01-11T22:29:23.0302843Z ok (3.310s)
2023-01-11T22:29:23.0302992Z 
2023-01-11T22:29:23.0303262Z ----------------------------------------------------------------------
2023-01-11T22:29:23.0303579Z Ran 12 tests in 33.288s
2023-01-11T22:29:23.0303743Z 
2023-01-11T22:29:23.0303835Z OK
2023-01-11T22:29:23.0303971Z 
2023-01-11T22:29:23.0304094Z Generating XML reports...
2023-01-11T22:29:23.0304772Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20230111222849.xml
2023-01-11T22:29:23.0305734Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20230111222849.xml
2023-01-11T22:29:23.0306777Z Generated XML report: test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20230111222849.xml
2023-01-11T22:29:23.0307278Z 
2023-01-11T22:29:23.0307672Z ##[endgroup]
2023-01-11T22:29:23.0308359Z FINISHED PRINTING LOG FILE of distributed/checkpoint/test_file_system_checkpoint_cpu (/var/lib/jenkins/workspace/test/test-reports/distributed-checkpoint-test_file_system_checkpoint_cpu_guxrkltv)
2023-01-11T22:29:23.0308781Z 
2023-01-11T22:29:23.0309054Z Running distributed/algorithms/test_join ... [2023-01-11 22:29:23.016115]
2023-01-11T22:29:23.0309738Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/algorithms/test_join.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:29:23.016500]
2023-01-11T22:30:00.3228195Z 
2023-01-11T22:30:00.3228672Z Expand the folded group to see the log file of distributed/algorithms/test_join
2023-01-11T22:30:00.3232955Z ##[group]PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-test_join_as0bq0c6)
2023-01-11T22:30:00.3233589Z 
2023-01-11T22:30:00.3233696Z Running tests...
2023-01-11T22:30:00.3234216Z ----------------------------------------------------------------------
2023-01-11T22:30:00.3235130Z Test results will be stored in test-reports/python-unittest/distributed.algorithms.test_join
2023-01-11T22:30:00.3235551Z   test_join_kwargs (__main__.TestJoin)
2023-01-11T22:30:00.3235984Z Tests passing keyword arguments to the context manager. ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:30:00.3236475Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65911
2023-01-11T22:30:00.3236940Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65912
2023-01-11T22:30:00.3237566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:30:00.3238036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:30:00.3238622Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:30:00.3239103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:30:00.3239686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:30:00.3240143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:30:00.3240950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:30:00.3241460Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:30:00.3242429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:30:00.3243362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:30:00.3244811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:30:00.3245322Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:30:00.3246028Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:30:00.3246746Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:30:00.3247151Z ok (5.305s)
2023-01-11T22:30:00.3247429Z   test_multiple_joinable_disable (__main__.TestJoin)
2023-01-11T22:30:00.3247921Z Tests ``enable=False`` for multiple :class:`Joinable` s. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 65991
2023-01-11T22:30:00.3248455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 65992
2023-01-11T22:30:00.3249074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:30:00.3249516Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:30:00.3250100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:30:00.3250577Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:30:00.3251172Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:30:00.3251607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:30:00.3252190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:30:00.3252670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:30:00.3253094Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:30:00.3253594Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:30:00.3254082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:30:00.3256095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:30:00.3257153Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:30:00.3258312Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:30:00.3258720Z ok (3.710s)
2023-01-11T22:30:00.3259162Z   test_multiple_joinables (__main__.TestJoin)
2023-01-11T22:30:00.3260462Z Tests the main hooks and post-hooks of multiple :class:`Joinable` s ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66071
2023-01-11T22:30:00.3261024Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66072
2023-01-11T22:30:00.3261649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:30:00.3262110Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:30:00.3262677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:30:00.3263312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:30:00.3263924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:30:00.3264435Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:30:00.3265016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:30:00.3265487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:30:00.3265929Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:30:00.3266405Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:30:00.3266897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:30:00.3267388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:30:00.3268039Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:30:00.3268744Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:30:00.3269145Z ok (3.710s)
2023-01-11T22:30:00.3269436Z   test_multiple_joinables_throw (__main__.TestJoin)
2023-01-11T22:30:00.3269898Z Tests ``throw_on_early_termination=True`` for multiple ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66151
2023-01-11T22:30:00.3270422Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66152
2023-01-11T22:30:00.3271038Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:30:00.3271494Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:30:00.3272062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:30:00.3272533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:30:00.3273113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:30:00.3273546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:30:00.3274125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:30:00.3274594Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:30:00.3275037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:30:00.3275514Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:30:00.3276004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:30:00.3276500Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:30:00.3277143Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:30:00.3277850Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:30:00.3278247Z ok (3.710s)
2023-01-11T22:30:00.3278523Z   test_single_joinable (__main__.TestJoin)
2023-01-11T22:30:00.3279121Z Tests the main hooks and post-hooks of a single :class:`Joinable` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66231
2023-01-11T22:30:00.3279657Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66232
2023-01-11T22:30:00.3280271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:30:00.3280784Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:30:00.3281364Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:30:00.3281917Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:30:00.3282505Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:30:00.3282936Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:30:00.3283515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:30:00.3283983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:30:00.3285270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:30:00.3286166Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:30:00.3287069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:30:00.3287967Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:30:00.3288861Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:30:00.3289562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:30:00.3289964Z ok (3.710s)
2023-01-11T22:30:00.3290253Z   test_single_joinable_disable (__main__.TestJoin)
2023-01-11T22:30:00.3290715Z Tests ``enable=False`` for a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66311
2023-01-11T22:30:00.3291236Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66312
2023-01-11T22:30:00.3291860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:30:00.3292319Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:30:00.3292882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:30:00.3293348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:30:00.3293933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:30:00.3294426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:30:00.3294991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:30:00.3295462Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:30:00.3295907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:30:00.3296404Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:30:00.3296876Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:30:00.3297366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:30:00.3298031Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:30:00.3298711Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:30:00.3299105Z ok (3.710s)
2023-01-11T22:30:00.3299398Z   test_single_joinable_main_hooks (__main__.TestJoin)
2023-01-11T22:30:00.3299874Z Tests the main hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66391
2023-01-11T22:30:00.3300481Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66392
2023-01-11T22:30:00.3301119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:30:00.3301644Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:30:00.3302220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:30:00.3302690Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:30:00.3303254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:30:00.3303698Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:30:00.3304278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:30:00.3304748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:30:00.3305170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:30:00.3305658Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:30:00.3306149Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:30:00.3306635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:30:00.3307282Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:30:00.3307972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:30:00.3308367Z ok (3.740s)
2023-01-11T22:30:00.3308641Z   test_single_joinable_post_hooks (__main__.TestJoin)
2023-01-11T22:30:00.3309244Z Tests the post-hooks of a single :class:`Joinable`. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66471
2023-01-11T22:30:00.3309762Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66472
2023-01-11T22:30:00.3310377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:30:00.3310808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:30:00.3311383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:30:00.3311855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:30:00.3312419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:30:00.3312862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:30:00.3313436Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:30:00.3313899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:30:00.3314322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:30:00.3314816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:30:00.3315304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:30:00.3315787Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:30:00.3316427Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:30:00.3317113Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:30:00.3317506Z ok (3.710s)
2023-01-11T22:30:00.3317833Z   test_single_joinable_throw (__main__.TestJoin)
2023-01-11T22:30:00.3318317Z Tests ``throw_on_early_termination=True`` for a single ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66551
2023-01-11T22:30:00.3318879Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66552
2023-01-11T22:30:00.3319493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:30:00.3319924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:30:00.3320502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:30:00.3320972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:30:00.3321534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:30:00.3321984Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:30:00.3322554Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:30:00.3323020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:30:00.3323441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:30:00.3323929Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:30:00.3325009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:30:00.3325511Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:30:00.3326173Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:30:00.3326871Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:30:00.3327266Z ok (3.710s)
2023-01-11T22:30:00.3327417Z 
2023-01-11T22:30:00.3327669Z ----------------------------------------------------------------------
2023-01-11T22:30:00.3328004Z Ran 9 tests in 35.016s
2023-01-11T22:30:00.3328167Z 
2023-01-11T22:30:00.3328260Z OK
2023-01-11T22:30:00.3328392Z 
2023-01-11T22:30:00.3328517Z Generating XML reports...
2023-01-11T22:30:00.3329078Z Generated XML report: test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20230111222924.xml
2023-01-11T22:30:00.3329412Z 
2023-01-11T22:30:00.3329858Z ##[endgroup]
2023-01-11T22:30:00.3330465Z FINISHED PRINTING LOG FILE of distributed/algorithms/test_join (/var/lib/jenkins/workspace/test/test-reports/distributed-algorithms-test_join_as0bq0c6)
2023-01-11T22:30:00.3330830Z 
2023-01-11T22:30:00.3331082Z Running distributed/test_c10d_spawn_nccl ... [2023-01-11 22:30:00.322980]
2023-01-11T22:30:00.3331797Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_spawn_nccl.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:30:00.323250]
2023-01-11T22:31:10.6496372Z 
2023-01-11T22:31:10.6497218Z Expand the folded group to see the log file of distributed/test_c10d_spawn_nccl
2023-01-11T22:31:10.6498685Z ##[group]PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_nccl_fab7rymy)
2023-01-11T22:31:10.6499863Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprtpq1qut
2023-01-11T22:31:10.6500963Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprtpq1qut/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6501523Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6501857Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:31:10.6502178Z <unittest.suite.TestSuite tests=[]>
2023-01-11T22:31:10.6504390Z <unittest.suite.TestSuite tests=[<__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_gather_base>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_all_to_all_single>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_allreduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_broadcast>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter>, <__main__.TestDistributedNNFunctionsNccl testMethod=test_reduce_scatter_non_contiguous>]>
2023-01-11T22:31:10.6506581Z test_all_gather (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T22:31:10.6507004Z test_all_gather_base (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T22:31:10.6507399Z test_all_to_all (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T22:31:10.6507809Z test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T22:31:10.6508228Z test_allreduce (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T22:31:10.6508613Z test_broadcast (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T22:31:10.6509010Z test_reduce (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T22:31:10.6511265Z test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T22:31:10.6512201Z test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl)
2023-01-11T22:31:10.6513583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6514374Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6515454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6516301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6517099Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpybt0h043
2023-01-11T22:31:10.6518221Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpybt0h043/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6518935Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6519145Z 
2023-01-11T22:31:10.6519253Z Running tests...
2023-01-11T22:31:10.6519680Z ----------------------------------------------------------------------
2023-01-11T22:31:10.6520226Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T22:31:10.6520788Z   test_all_gather (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66700
2023-01-11T22:31:10.6521334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66701
2023-01-11T22:31:10.6521947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6522400Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6522966Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6523439Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6524019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6524823Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6525416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6525886Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6526359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpocyrlddc
2023-01-11T22:31:10.6526887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpocyrlddc/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6527602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp685schp_
2023-01-11T22:31:10.6528162Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp685schp_/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6528651Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6529060Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:10.6529710Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:10.6530114Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6530500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:10.6530987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:10.6531663Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:10.6532369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:10.6532744Z ok (4.224s)
2023-01-11T22:31:10.6532893Z 
2023-01-11T22:31:10.6533166Z ----------------------------------------------------------------------
2023-01-11T22:31:10.6533494Z Ran 1 test in 4.224s
2023-01-11T22:31:10.6533655Z 
2023-01-11T22:31:10.6533731Z OK
2023-01-11T22:31:10.6533864Z 
2023-01-11T22:31:10.6533987Z Generating XML reports...
2023-01-11T22:31:10.6534630Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111223006.xml
2023-01-11T22:31:10.6535421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6535862Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6536447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6536918Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6537373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0mcy9pno
2023-01-11T22:31:10.6537922Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0mcy9pno/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6538352Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6538548Z 
2023-01-11T22:31:10.6538654Z Running tests...
2023-01-11T22:31:10.6539040Z ----------------------------------------------------------------------
2023-01-11T22:31:10.6539579Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T22:31:10.6540161Z   test_all_gather_base (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66819
2023-01-11T22:31:10.6540697Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66820
2023-01-11T22:31:10.6541314Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6541769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6542353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6542807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6543384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6543832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6544387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6544855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6545393Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcrgfhwdp
2023-01-11T22:31:10.6545948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcrgfhwdp/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6546519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppj3520ld
2023-01-11T22:31:10.6547052Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppj3520ld/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6547481Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6547890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:10.6548366Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:10.6548770Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6549173Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:10.6549648Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:10.6550322Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:10.6551023Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:10.6551945Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:31:10.6552559Z   warnings.warn(
2023-01-11T22:31:10.6553305Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:2533: UserWarning: torch.distributed._all_gather_base is a private function and will be deprecated. Please use torch.distributed.all_gather_into_tensor instead.
2023-01-11T22:31:10.6553845Z   warnings.warn(
2023-01-11T22:31:10.6554618Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:3001: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2023-01-11T22:31:10.6555174Z   warnings.warn(
2023-01-11T22:31:10.6555912Z /opt/conda/lib/python3.10/site-packages/torch/distributed/distributed_c10d.py:3001: UserWarning: torch.distributed._reduce_scatter_base is a private function and will be deprecated. Please use torch.distributed.reduce_scatter_tensor instead.
2023-01-11T22:31:10.6556465Z   warnings.warn(
2023-01-11T22:31:10.6556700Z ok (4.125s)
2023-01-11T22:31:10.6556847Z 
2023-01-11T22:31:10.6557099Z ----------------------------------------------------------------------
2023-01-11T22:31:10.6557427Z Ran 1 test in 4.125s
2023-01-11T22:31:10.6557588Z 
2023-01-11T22:31:10.6557682Z OK
2023-01-11T22:31:10.6557814Z 
2023-01-11T22:31:10.6557938Z Generating XML reports...
2023-01-11T22:31:10.6558573Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111223014.xml
2023-01-11T22:31:10.6559318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6559776Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6560339Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6560813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6561278Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2qn_lsgz
2023-01-11T22:31:10.6561815Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2qn_lsgz/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6562227Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6562423Z 
2023-01-11T22:31:10.6562531Z Running tests...
2023-01-11T22:31:10.6563007Z ----------------------------------------------------------------------
2023-01-11T22:31:10.6563543Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T22:31:10.6564170Z   test_all_to_all (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 66938
2023-01-11T22:31:10.6564993Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 66939
2023-01-11T22:31:10.6565618Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6566054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6566632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6567104Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6567688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6568115Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6568693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6585415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6585984Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfo40gtbu
2023-01-11T22:31:10.6586526Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfo40gtbu/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6587069Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_no6h07r
2023-01-11T22:31:10.6587615Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_no6h07r/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6588028Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6588358Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6588766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:10.6589243Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:10.6589740Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:10.6590230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:10.6590938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:10.6591627Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:10.6592282Z ok (4.124s)
2023-01-11T22:31:10.6592434Z 
2023-01-11T22:31:10.6592711Z ----------------------------------------------------------------------
2023-01-11T22:31:10.6593045Z Ran 1 test in 4.125s
2023-01-11T22:31:10.6593191Z 
2023-01-11T22:31:10.6593285Z OK
2023-01-11T22:31:10.6593420Z 
2023-01-11T22:31:10.6593544Z Generating XML reports...
2023-01-11T22:31:10.6594195Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111223021.xml
2023-01-11T22:31:10.6594929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6595388Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6595973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6596444Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6596891Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp_1nojea
2023-01-11T22:31:10.6597583Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp_1nojea/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6598035Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6598233Z 
2023-01-11T22:31:10.6598413Z Running tests...
2023-01-11T22:31:10.6598829Z ----------------------------------------------------------------------
2023-01-11T22:31:10.6599373Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T22:31:10.6599957Z   test_all_to_all_single (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67059
2023-01-11T22:31:10.6600484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67060
2023-01-11T22:31:10.6601075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6601515Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6602096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6602555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6603142Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6603586Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6604135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6604967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6605438Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ddwjzs6
2023-01-11T22:31:10.6605982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ddwjzs6/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6606505Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ewq7lxx
2023-01-11T22:31:10.6607046Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ewq7lxx/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6607474Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6607783Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6608187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:10.6608680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:10.6609170Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:10.6609635Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:10.6610311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:10.6611012Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:10.6611395Z ok (4.124s)
2023-01-11T22:31:10.6611543Z 
2023-01-11T22:31:10.6611808Z ----------------------------------------------------------------------
2023-01-11T22:31:10.6612139Z Ran 1 test in 4.125s
2023-01-11T22:31:10.6612300Z 
2023-01-11T22:31:10.6612394Z OK
2023-01-11T22:31:10.6612508Z 
2023-01-11T22:31:10.6612632Z Generating XML reports...
2023-01-11T22:31:10.6613280Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111223028.xml
2023-01-11T22:31:10.6614023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6614463Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6615043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6615613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6616097Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7v5_zy8y
2023-01-11T22:31:10.6616688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7v5_zy8y/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6617116Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6617313Z 
2023-01-11T22:31:10.6617420Z Running tests...
2023-01-11T22:31:10.6617814Z ----------------------------------------------------------------------
2023-01-11T22:31:10.6618356Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T22:31:10.6618933Z   test_allreduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67180
2023-01-11T22:31:10.6619474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67181
2023-01-11T22:31:10.6620070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6620522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6621101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6621576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6622135Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6622584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6623153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6623605Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6624079Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjyvidzpv
2023-01-11T22:31:10.6624625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjyvidzpv/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6625163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp96stqdg2
2023-01-11T22:31:10.6625684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp96stqdg2/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6626114Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6626524Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:10.6626998Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:10.6627406Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6627809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:10.6628299Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:10.6628954Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:10.6629649Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:10.6630049Z ok (4.124s)
2023-01-11T22:31:10.6630196Z 
2023-01-11T22:31:10.6630445Z ----------------------------------------------------------------------
2023-01-11T22:31:10.6630774Z Ran 1 test in 4.124s
2023-01-11T22:31:10.6630934Z 
2023-01-11T22:31:10.6631027Z OK
2023-01-11T22:31:10.6631159Z 
2023-01-11T22:31:10.6631283Z Generating XML reports...
2023-01-11T22:31:10.6631913Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111223036.xml
2023-01-11T22:31:10.6632662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6633200Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6633796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6634313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6634786Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjfsndl1c
2023-01-11T22:31:10.6635378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjfsndl1c/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6635798Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6635993Z 
2023-01-11T22:31:10.6636102Z Running tests...
2023-01-11T22:31:10.6636516Z ----------------------------------------------------------------------
2023-01-11T22:31:10.6637040Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T22:31:10.6637631Z   test_broadcast (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67299
2023-01-11T22:31:10.6638179Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67300
2023-01-11T22:31:10.6638796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6639235Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6639809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6640282Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6640863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6641290Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6641859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6642323Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6642771Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfqif3pxf
2023-01-11T22:31:10.6643315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfqif3pxf/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6643861Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxxlmot2y
2023-01-11T22:31:10.6644669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxxlmot2y/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6645091Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6645413Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6645819Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:10.6646274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:10.6646770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:10.6647275Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:10.6647949Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:10.6648626Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:10.6649018Z ok (4.124s)
2023-01-11T22:31:10.6649167Z 
2023-01-11T22:31:10.6649435Z ----------------------------------------------------------------------
2023-01-11T22:31:10.6649761Z Ran 1 test in 4.124s
2023-01-11T22:31:10.6649906Z 
2023-01-11T22:31:10.6649999Z OK
2023-01-11T22:31:10.6650133Z 
2023-01-11T22:31:10.6650259Z Generating XML reports...
2023-01-11T22:31:10.6650994Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111223043.xml
2023-01-11T22:31:10.6651745Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6652274Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6652852Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6653327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6653776Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp72g1l4e1
2023-01-11T22:31:10.6654317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp72g1l4e1/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6654745Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6654942Z 
2023-01-11T22:31:10.6655032Z Running tests...
2023-01-11T22:31:10.6655440Z ----------------------------------------------------------------------
2023-01-11T22:31:10.6655980Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T22:31:10.6656556Z   test_reduce (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67418
2023-01-11T22:31:10.6657088Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67419
2023-01-11T22:31:10.6657700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6658154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6658712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6659180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6659768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6660217Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6660769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6661241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6661711Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfoor71cy
2023-01-11T22:31:10.6662258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfoor71cy/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6662779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpllavqsc_
2023-01-11T22:31:10.6663325Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpllavqsc_/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6663756Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6664154Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:10.6664649Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:10.6665059Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6665449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:10.6665940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:10.6666605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:10.6667297Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:10.6667670Z ok (4.124s)
2023-01-11T22:31:10.6667816Z 
2023-01-11T22:31:10.6668083Z ----------------------------------------------------------------------
2023-01-11T22:31:10.6668410Z Ran 1 test in 4.124s
2023-01-11T22:31:10.6668570Z 
2023-01-11T22:31:10.6668732Z OK
2023-01-11T22:31:10.6668866Z 
2023-01-11T22:31:10.6668997Z Generating XML reports...
2023-01-11T22:31:10.6669677Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111223051.xml
2023-01-11T22:31:10.6670543Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6671011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6671623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6672124Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6672616Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8im00bfg
2023-01-11T22:31:10.6673167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8im00bfg/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6673622Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6673830Z 
2023-01-11T22:31:10.6673945Z Running tests...
2023-01-11T22:31:10.6674362Z ----------------------------------------------------------------------
2023-01-11T22:31:10.6674940Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T22:31:10.6675559Z   test_reduce_scatter (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67537
2023-01-11T22:31:10.6676141Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67538
2023-01-11T22:31:10.6676769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6677253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6677870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6678352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6678967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6679450Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6680061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6680542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6681032Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwofn_ad5
2023-01-11T22:31:10.6681600Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwofn_ad5/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6682163Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmm6pd2tm
2023-01-11T22:31:10.6682716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmm6pd2tm/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6683169Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6683602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:10.6684103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:10.6684890Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6685318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:10.6685834Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:10.6686732Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:10.6687472Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:10.6687984Z ok (4.025s)
2023-01-11T22:31:10.6688149Z 
2023-01-11T22:31:10.6688421Z ----------------------------------------------------------------------
2023-01-11T22:31:10.6688838Z Ran 1 test in 4.025s
2023-01-11T22:31:10.6689008Z 
2023-01-11T22:31:10.6689103Z OK
2023-01-11T22:31:10.6689244Z 
2023-01-11T22:31:10.6689371Z Generating XML reports...
2023-01-11T22:31:10.6690036Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111223058.xml
2023-01-11T22:31:10.6690824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6691305Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6691897Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6692405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6692901Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg7q_kvix
2023-01-11T22:31:10.6693464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg7q_kvix/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6693901Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6694108Z 
2023-01-11T22:31:10.6694219Z Running tests...
2023-01-11T22:31:10.6694649Z ----------------------------------------------------------------------
2023-01-11T22:31:10.6695205Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_spawn_nccl
2023-01-11T22:31:10.6695844Z   test_reduce_scatter_non_contiguous (__main__.TestDistributedNNFunctionsNccl) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67656
2023-01-11T22:31:10.6696453Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67657
2023-01-11T22:31:10.6697112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6697576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6698188Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6698692Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6699309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:31:10.6699769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:31:10.6700377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:31:10.6700878Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:31:10.6701346Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgv9ueig0
2023-01-11T22:31:10.6701921Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgv9ueig0/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6702489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7dng8yx9
2023-01-11T22:31:10.6703057Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7dng8yx9/_remote_module_non_scriptable.py
2023-01-11T22:31:10.6703487Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6703825Z INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:31:10.6704250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:31:10.6704750Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:31:10.6705263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:31:10.6705782Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:31:10.6706546Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:10.6707284Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:31:10.6707773Z ok (4.125s)
2023-01-11T22:31:10.6707927Z 
2023-01-11T22:31:10.6708211Z ----------------------------------------------------------------------
2023-01-11T22:31:10.6708538Z Ran 1 test in 4.125s
2023-01-11T22:31:10.6708709Z 
2023-01-11T22:31:10.6708804Z OK
2023-01-11T22:31:10.6708943Z 
2023-01-11T22:31:10.6709073Z Generating XML reports...
2023-01-11T22:31:10.6709752Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111223105.xml
2023-01-11T22:31:10.6710165Z 
2023-01-11T22:31:10.6710564Z ##[endgroup]
2023-01-11T22:31:10.6711195Z FINISHED PRINTING LOG FILE of distributed/test_c10d_spawn_nccl (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_spawn_nccl_fab7rymy)
2023-01-11T22:31:10.6711565Z 
2023-01-11T22:31:10.6711855Z Running distributed/fsdp/test_fsdp_grad_acc ... [2023-01-11 22:31:10.649861]
2023-01-11T22:31:10.6712573Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_grad_acc.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:31:10.650127]
2023-01-11T22:32:10.0484737Z 
2023-01-11T22:32:10.0485738Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_grad_acc
2023-01-11T22:32:10.0486725Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_grad_acc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_grad_acc_cghlm_kg)
2023-01-11T22:32:10.0487330Z 
2023-01-11T22:32:10.0487436Z Running tests...
2023-01-11T22:32:10.0487995Z ----------------------------------------------------------------------
2023-01-11T22:32:10.0488965Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc
2023-01-11T22:32:10.0496855Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_sharding_strategy_ShardingStrategy_FULL_SHARD_use_orig_params_False (__main__.TestGradAcc)
2023-01-11T22:32:10.0497463Z Tests gradient accumulation. ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:32:10.0497927Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67775
2023-01-11T22:32:10.0498391Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67776
2023-01-11T22:32:10.0499080Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0499529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0500119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0500606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0501202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0501641Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0502232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0502710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0503163Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:32:10.0503664Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:32:10.0504344Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0505294Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0505829Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:32:10.0506309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:32:10.0507432Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0508751Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0510019Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0511268Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0512499Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0513745Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0514986Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0516219Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0516840Z dist init r=1, world=2
2023-01-11T22:32:10.0517102Z dist init r=0, world=2
2023-01-11T22:32:10.0517347Z ok (6.125s)
2023-01-11T22:32:10.0517838Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_sharding_strategy_ShardingStrategy_FULL_SHARD_use_orig_params_True (__main__.TestGradAcc)
2023-01-11T22:32:10.0518491Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67858
2023-01-11T22:32:10.0518990Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67859
2023-01-11T22:32:10.0519616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0520055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0520638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0521187Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0521773Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0522285Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0522869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0523343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0523786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:32:10.0524641Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:32:10.0525336Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0526039Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0526552Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:32:10.0527034Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:32:10.0527394Z dist init r=1, world=2
2023-01-11T22:32:10.0527631Z dist init r=0, world=2
2023-01-11T22:32:10.0527871Z ok (4.812s)
2023-01-11T22:32:10.0528373Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_sharding_strategy_ShardingStrategy_NO_SHARD_use_orig_params_False (__main__.TestGradAcc)
2023-01-11T22:32:10.0529013Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 67941
2023-01-11T22:32:10.0529492Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 67942
2023-01-11T22:32:10.0530120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0530580Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0531146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0531631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0532221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0532671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0533237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0533709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0534162Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:32:10.0534665Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:32:10.0535315Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0536020Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0536550Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:32:10.0537011Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:32:10.0538069Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0539436Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0540751Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0541983Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0543229Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0544461Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0545699Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0546907Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0547528Z dist init r=1, world=2
2023-01-11T22:32:10.0547781Z dist init r=0, world=2
2023-01-11T22:32:10.0548022Z ok (4.412s)
2023-01-11T22:32:10.0548506Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_sharding_strategy_ShardingStrategy_NO_SHARD_use_orig_params_True (__main__.TestGradAcc)
2023-01-11T22:32:10.0549150Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68024
2023-01-11T22:32:10.0549648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68025
2023-01-11T22:32:10.0550272Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0550708Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0551291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0551903Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0552479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0552932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0553508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0553978Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0554419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:32:10.0554975Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:32:10.0555662Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0556415Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0556924Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:32:10.0557401Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:32:10.0558406Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0559662Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0560914Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0562136Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0563391Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0564923Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0566178Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0567413Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0568656Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0569882Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0571208Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0572506Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0573722Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0574946Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0576186Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0577423Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0578656Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0579896Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0581129Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0582359Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0582952Z dist init r=0, world=2
2023-01-11T22:32:10.0583205Z dist init r=1, world=2
2023-01-11T22:32:10.0583447Z ok (4.813s)
2023-01-11T22:32:10.0583937Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_use_orig_params_False (__main__.TestGradAcc)
2023-01-11T22:32:10.0584584Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68107
2023-01-11T22:32:10.0585084Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68108
2023-01-11T22:32:10.0585761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0586209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0586794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0587338Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0587924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0588355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0588931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0589408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0589849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:32:10.0590357Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:32:10.0591018Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0591723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0592235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:32:10.0592713Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:32:10.0593715Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0594969Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0596221Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0597454Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0598686Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0599915Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0601161Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0602451Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0603105Z dist init r=0, world=2
2023-01-11T22:32:10.0603360Z dist init r=1, world=2
2023-01-11T22:32:10.0603583Z ok (4.412s)
2023-01-11T22:32:10.0604096Z   test_grad_acc_configs_[(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3)]_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_use_orig_params_True (__main__.TestGradAcc)
2023-01-11T22:32:10.0605088Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68190
2023-01-11T22:32:10.0605573Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68191
2023-01-11T22:32:10.0606203Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0606661Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0607252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0607710Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0608294Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0608747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0609326Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0609776Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0610238Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:32:10.0610743Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:32:10.0611393Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0612096Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0612632Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:32:10.0613109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:32:10.0613450Z dist init r=0, world=2
2023-01-11T22:32:10.0613704Z dist init r=1, world=2
2023-01-11T22:32:10.0613944Z ok (4.812s)
2023-01-11T22:32:10.0614431Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_sharding_strategy_ShardingStrategy_FULL_SHARD_use_orig_params_False (__main__.TestGradAcc)
2023-01-11T22:32:10.0615076Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68273
2023-01-11T22:32:10.0615575Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68274
2023-01-11T22:32:10.0616192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0616630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0617209Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0617683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0618265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0618778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0619378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0619916Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0620355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:32:10.0620859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:32:10.0621528Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0622220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0622729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:32:10.0623207Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:32:10.0624217Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0625479Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0626725Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0627967Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0629196Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0630441Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0631675Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0632918Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0633528Z dist init r=0, world=2
2023-01-11T22:32:10.0633784Z dist init r=1, world=2
2023-01-11T22:32:10.0634006Z ok (4.512s)
2023-01-11T22:32:10.0634561Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_sharding_strategy_ShardingStrategy_FULL_SHARD_use_orig_params_True (__main__.TestGradAcc)
2023-01-11T22:32:10.0635215Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68356
2023-01-11T22:32:10.0635746Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68357
2023-01-11T22:32:10.0636369Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0636829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0637418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0637926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0638515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0638971Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0639531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0640011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0640470Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:32:10.0640974Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:32:10.0641616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0642313Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0642843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:32:10.0643328Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:32:10.0643670Z dist init r=1, world=2
2023-01-11T22:32:10.0643924Z dist init r=0, world=2
2023-01-11T22:32:10.0644167Z ok (4.812s)
2023-01-11T22:32:10.0644935Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_sharding_strategy_ShardingStrategy_NO_SHARD_use_orig_params_False (__main__.TestGradAcc)
2023-01-11T22:32:10.0645572Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68439
2023-01-11T22:32:10.0646070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68440
2023-01-11T22:32:10.0646699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0647135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0647717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0648189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0648758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0649205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0649778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0650246Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0650684Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:32:10.0651187Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:32:10.0651930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0652648Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0653227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:32:10.0653706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:32:10.0654717Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0655972Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0657220Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0658438Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0659689Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0660920Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0662150Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0663388Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0663998Z dist init r=1, world=2
2023-01-11T22:32:10.0664235Z dist init r=0, world=2
2023-01-11T22:32:10.0664479Z ok (4.411s)
2023-01-11T22:32:10.0664978Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_sharding_strategy_ShardingStrategy_NO_SHARD_use_orig_params_True (__main__.TestGradAcc)
2023-01-11T22:32:10.0665618Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68522
2023-01-11T22:32:10.0666100Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68523
2023-01-11T22:32:10.0666720Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0667176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0667815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0668283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0668924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0669380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0669940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0670417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0670877Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:32:10.0671376Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:32:10.0672030Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0672737Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0673269Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:32:10.0673751Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:32:10.0674738Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0675994Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0677248Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0678483Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0679728Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0680957Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0682200Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0683460Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0685103Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0686453Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0687738Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0688974Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0690212Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0691447Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0692704Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0693940Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0695155Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0696388Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0697620Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0698915Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0699541Z dist init r=1, world=2
2023-01-11T22:32:10.0699796Z dist init r=0, world=2
2023-01-11T22:32:10.0700082Z ok (4.812s)
2023-01-11T22:32:10.0700591Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_use_orig_params_False (__main__.TestGradAcc)
2023-01-11T22:32:10.0701237Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68605
2023-01-11T22:32:10.0701720Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68606
2023-01-11T22:32:10.0702343Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0702801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0703387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0703845Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0704432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0704880Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0705458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0705911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0706367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:32:10.0706868Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:32:10.0707520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0708220Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0708752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:32:10.0709231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:32:10.0710219Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0711485Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0712733Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0713982Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0715270Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0716544Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0717834Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0719077Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:32:10.0719667Z dist init r=0, world=2
2023-01-11T22:32:10.0719919Z dist init r=1, world=2
2023-01-11T22:32:10.0720167Z ok (4.412s)
2023-01-11T22:32:10.0720656Z   test_grad_acc_configs_[(use_no_sync=True,num_iters=3),(use_no_sync=False,num_iters=3),(use_no_sync=True,num_iters=3)]_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP_use_orig_params_True (__main__.TestGradAcc)
2023-01-11T22:32:10.0721306Z Tests gradient accumulation. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68688
2023-01-11T22:32:10.0721809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68689
2023-01-11T22:32:10.0722424Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0722863Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0723453Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0723935Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0725073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:32:10.0725519Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:32:10.0726105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:32:10.0726581Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:32:10.0727023Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:32:10.0727525Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:32:10.0728223Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0728927Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:32:10.0729441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:32:10.0729919Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:32:10.0730271Z dist init r=1, world=2
2023-01-11T22:32:10.0730524Z dist init r=0, world=2
2023-01-11T22:32:10.0730745Z ok (4.712s)
2023-01-11T22:32:10.0730897Z 
2023-01-11T22:32:10.0731168Z ----------------------------------------------------------------------
2023-01-11T22:32:10.0731504Z Ran 12 tests in 57.057s
2023-01-11T22:32:10.0731650Z 
2023-01-11T22:32:10.0731745Z OK
2023-01-11T22:32:10.0731878Z 
2023-01-11T22:32:10.0732003Z Generating XML reports...
2023-01-11T22:32:10.0732681Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20230111223112.xml
2023-01-11T22:32:10.0733039Z 
2023-01-11T22:32:10.0733365Z ##[endgroup]
2023-01-11T22:32:10.0733976Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_grad_acc (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_grad_acc_cghlm_kg)
2023-01-11T22:32:10.0734425Z 
2023-01-11T22:32:10.0734711Z Running distributed/_tensor/test_tensor_ops ... [2023-01-11 22:32:10.048787]
2023-01-11T22:32:10.0735398Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/_tensor/test_tensor_ops.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:32:10.049130]
2023-01-11T22:33:13.7840530Z 
2023-01-11T22:33:13.7841354Z Expand the folded group to see the log file of distributed/_tensor/test_tensor_ops
2023-01-11T22:33:13.7842319Z ##[group]PRINTING LOG FILE of distributed/_tensor/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tensor_ops_w94xcsxm)
2023-01-11T22:33:13.7842682Z 
2023-01-11T22:33:13.7842810Z Running tests...
2023-01-11T22:33:13.7849173Z ----------------------------------------------------------------------
2023-01-11T22:33:13.7849821Z Test results will be stored in test-reports/python-unittest/distributed._tensor.test_tensor_ops
2023-01-11T22:33:13.7850362Z   test_aten_contiguous (__main__.DistTensorOpsTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:33:13.7850843Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68806
2023-01-11T22:33:13.7851279Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68807
2023-01-11T22:33:13.7851931Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7852390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7852980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7853448Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7854035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7854499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7855076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7855696Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7856450Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:13.7857365Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:33:13.7858324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:13.7859363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:33:13.7860081Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7860786Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7861181Z ok (4.777s)
2023-01-11T22:33:13.7861598Z   test_clone (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68887
2023-01-11T22:33:13.7862314Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68888
2023-01-11T22:33:13.7863145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7863970Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7865584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7866575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7867602Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7868763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7869955Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7870445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7870892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:13.7871389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:33:13.7871859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:13.7872358Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:33:13.7873026Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7873728Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7874107Z ok (3.209s)
2023-01-11T22:33:13.7874532Z   test_contiguous (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 68968
2023-01-11T22:33:13.7875048Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 68969
2023-01-11T22:33:13.7875644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7876095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7876680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7877159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7877728Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7878177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7878754Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7879204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7879646Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:13.7880141Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:33:13.7880631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:13.7881103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:33:13.7881764Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7882462Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7882857Z ok (3.209s)
2023-01-11T22:33:13.7883252Z   test_detach (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69051
2023-01-11T22:33:13.7883763Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69052
2023-01-11T22:33:13.7884923Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7885379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7886086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7886578Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7887231Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7887658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7888230Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7888699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7889123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:13.7889619Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:33:13.7890115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:13.7890604Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:33:13.7891248Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7891938Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7892333Z ok (3.209s)
2023-01-11T22:33:13.7892750Z   test_empty_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69132
2023-01-11T22:33:13.7893248Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69133
2023-01-11T22:33:13.7893863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7894315Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7894878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7895348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7895932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7896375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7896934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7897398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7897840Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:13.7898316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:33:13.7898809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:13.7899294Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:33:13.7899957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7900635Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7901031Z ok (3.209s)
2023-01-11T22:33:13.7901455Z   test_fill_inplace (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69211
2023-01-11T22:33:13.7901976Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69212
2023-01-11T22:33:13.7902572Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7903087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7903679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7904195Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7904780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7905230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7905801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7906256Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7906694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:13.7907172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:13.7907668Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:33:13.7908148Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:33:13.7908818Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7909511Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7909888Z ok (3.209s)
2023-01-11T22:33:13.7910322Z   test_fill_inplace_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69290
2023-01-11T22:33:13.7910854Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69291
2023-01-11T22:33:13.7911464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7911901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7912474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7912950Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7913515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7913962Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7914537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7915002Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7915424Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:13.7915920Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:33:13.7916407Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:13.7916895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:33:13.7917533Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7918226Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7918621Z ok (3.209s)
2023-01-11T22:33:13.7919020Z   test_full_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69369
2023-01-11T22:33:13.7919532Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69370
2023-01-11T22:33:13.7920199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7920660Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7921222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7921744Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7922325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7922769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7923324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7923789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7924722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:13.7925252Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:33:13.7925736Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:13.7926230Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:33:13.7926901Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7927579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7927973Z ok (3.209s)
2023-01-11T22:33:13.7928388Z   test_index (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69448
2023-01-11T22:33:13.7928878Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69449
2023-01-11T22:33:13.7929494Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7929946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7930527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7930982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7931561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7932009Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7932584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7933034Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7933481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:13.7933972Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:33:13.7934447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:13.7934939Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:33:13.7935604Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7936296Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7936671Z ok (11.226s)
2023-01-11T22:33:13.7937088Z   test_inplace_op (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69559
2023-01-11T22:33:13.7937606Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69560
2023-01-11T22:33:13.7938303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7938764Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7939444Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7939915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7940479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7940978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7941553Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7942020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7942447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:13.7942943Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:33:13.7943433Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:13.7943901Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:33:13.7944562Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7945256Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7945650Z ok (3.310s)
2023-01-11T22:33:13.7946050Z   test_ones_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69640
2023-01-11T22:33:13.7946563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69641
2023-01-11T22:33:13.7947174Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7947629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7948190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7948660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7949243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7949671Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7950240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7950708Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7951151Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:13.7951622Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:33:13.7952109Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:13.7952600Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:33:13.7953237Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7953932Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7954327Z ok (3.209s)
2023-01-11T22:33:13.7954757Z   test_ones_like_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69719
2023-01-11T22:33:13.7955336Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69720
2023-01-11T22:33:13.7955963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7956471Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7957054Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7957509Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7958086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7958535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7959087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7959555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7959994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:13.7960490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:33:13.7960958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:13.7961447Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:33:13.7962110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7962799Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7963175Z ok (3.309s)
2023-01-11T22:33:13.7963601Z   test_op_out_variant (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69798
2023-01-11T22:33:13.7964119Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69799
2023-01-11T22:33:13.7965253Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7965713Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7966293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7966767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7967332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7967781Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7968353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7968809Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7969254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:13.7969752Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:33:13.7970238Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:13.7970705Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:33:13.7971369Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7972065Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7972461Z ok (3.309s)
2023-01-11T22:33:13.7972962Z   test_zero_inplace (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69879
2023-01-11T22:33:13.7973495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69880
2023-01-11T22:33:13.7974173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7974611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7975186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7975662Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7976243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7976674Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7977256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7977723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7978145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:13.7978625Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:13.7979114Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:33:13.7979612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:33:13.7980260Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7980957Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7981351Z ok (3.209s)
2023-01-11T22:33:13.7981774Z   test_zeros_like (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 69958
2023-01-11T22:33:13.7982272Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 69959
2023-01-11T22:33:13.7982890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7983343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7983902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7984374Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7984951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7985398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7985957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7986426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7986865Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:13.7987337Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:33:13.7987821Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:13.7988307Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:33:13.7988968Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7989647Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7990041Z ok (3.309s)
2023-01-11T22:33:13.7990534Z   test_zeros_like_partial_sum (__main__.DistTensorOpsTest) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70037
2023-01-11T22:33:13.7991110Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70038
2023-01-11T22:33:13.7991710Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7992164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7992742Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7993199Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7993776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:33:13.7994226Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:33:13.7994801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:33:13.7995277Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:33:13.7995721Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:33:13.7996214Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:33:13.7996704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:33:13.7997168Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:33:13.7997827Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7998520Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:33:13.7998914Z ok (3.309s)
2023-01-11T22:33:13.7999044Z 
2023-01-11T22:33:13.7999317Z ----------------------------------------------------------------------
2023-01-11T22:33:13.7999654Z Ran 16 tests in 61.432s
2023-01-11T22:33:13.7999820Z 
2023-01-11T22:33:13.7999911Z OK
2023-01-11T22:33:13.8000044Z 
2023-01-11T22:33:13.8000152Z Generating XML reports...
2023-01-11T22:33:13.8000749Z Generated XML report: test-reports/python-unittest/distributed._tensor.test_tensor_ops/TEST-DistTensorOpsTest-20230111223211.xml
2023-01-11T22:33:13.8001105Z 
2023-01-11T22:33:13.8001497Z ##[endgroup]
2023-01-11T22:33:13.8002079Z FINISHED PRINTING LOG FILE of distributed/_tensor/test_tensor_ops (/var/lib/jenkins/workspace/test/test-reports/distributed-_tensor-test_tensor_ops_w94xcsxm)
2023-01-11T22:33:13.8002434Z 
2023-01-11T22:33:13.8002707Z Running distributed/fsdp/test_fsdp_comm_hooks ... [2023-01-11 22:33:13.784270]
2023-01-11T22:33:13.8003402Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_comm_hooks.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:33:13.784594]
2023-01-11T22:34:57.3898968Z 
2023-01-11T22:34:57.3902069Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_comm_hooks
2023-01-11T22:34:57.3903368Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_hooks_tcezadhs)
2023-01-11T22:34:57.3903762Z 
2023-01-11T22:34:57.3903876Z Running tests...
2023-01-11T22:34:57.3904590Z ----------------------------------------------------------------------
2023-01-11T22:34:57.3910954Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks
2023-01-11T22:34:57.3911802Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:34:57.3912832Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70151
2023-01-11T22:34:57.3913334Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70152
2023-01-11T22:34:57.3914234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.3914848Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.3915748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.3916241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.3917090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.3917554Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.3918404Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.3918899Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.3919474Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.3920152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.3921040Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.3921809Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.3922587Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.3923082Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.3923496Z dist init r=1, world=2
2023-01-11T22:34:57.3923942Z dist init r=0, world=2
2023-01-11T22:34:57.3924545Z ok (5.422s)
2023-01-11T22:34:57.3925348Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70234
2023-01-11T22:34:57.3925991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70235
2023-01-11T22:34:57.3926881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.3927353Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.3928232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.3928704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.3929577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.3930054Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.3930896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.3931583Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.3932308Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.3932829Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.3933770Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.3934477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.3935274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.3935902Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.3936511Z dist init r=1, world=2
2023-01-11T22:34:57.3936746Z dist init r=0, world=2
2023-01-11T22:34:57.3937090Z ok (3.810s)
2023-01-11T22:34:57.3937874Z   test_bf16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70317
2023-01-11T22:34:57.3938484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70318
2023-01-11T22:34:57.3939381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.3939844Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.3940734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.3941204Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.3942073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.3942545Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.3943373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.3943861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.3944468Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.3945071Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.3945988Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.3946726Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.3947513Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.3948004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.3948430Z dist init r=0, world=2
2023-01-11T22:34:57.3948845Z dist init r=1, world=2
2023-01-11T22:34:57.3949089Z ok (3.810s)
2023-01-11T22:34:57.3949593Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70400
2023-01-11T22:34:57.3950455Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70401
2023-01-11T22:34:57.3951184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.3951793Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.3952395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.3953094Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.3953693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.3954378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.3954963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.3955686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.3956149Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.3956830Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.3957628Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.3958625Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.3959299Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.3959976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.3960332Z dist init r=1, world=2
2023-01-11T22:34:57.3960598Z dist init r=0, world=2
2023-01-11T22:34:57.3961017Z ok (3.810s)
2023-01-11T22:34:57.3961569Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70483
2023-01-11T22:34:57.3962380Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70484
2023-01-11T22:34:57.3963030Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.3963668Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.3964696Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.3965436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.3966010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.3966538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.3967305Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.3967781Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.3968228Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.3968731Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.3969681Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.3970366Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.3970895Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.3971373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.3971734Z dist init r=0, world=2
2023-01-11T22:34:57.3971972Z dist init r=1, world=2
2023-01-11T22:34:57.3972215Z ok (3.810s)
2023-01-11T22:34:57.3972738Z   test_bf16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70566
2023-01-11T22:34:57.3973609Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70567
2023-01-11T22:34:57.3974248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.3974706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.3975285Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.3975747Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.3976334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.3976782Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.3977468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.3977939Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.3978396Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.3978961Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.3979614Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.3980374Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.3981132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.3981613Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.3981955Z dist init r=0, world=2
2023-01-11T22:34:57.3982215Z dist init r=1, world=2
2023-01-11T22:34:57.3982457Z ok (3.810s)
2023-01-11T22:34:57.3982859Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:34:57.3983629Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70649
2023-01-11T22:34:57.3984178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70650
2023-01-11T22:34:57.3984789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.3985224Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.3985805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.3986285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.3986859Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.3987314Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.3987900Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.3988372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.3988815Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.3989316Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.3989978Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.3990676Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.3991184Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.3991664Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.3992024Z dist init r=0, world=2
2023-01-11T22:34:57.3992256Z dist init r=1, world=2
2023-01-11T22:34:57.3992496Z ok (3.711s)
2023-01-11T22:34:57.3992912Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:34:57.3993664Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70732
2023-01-11T22:34:57.3994392Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70733
2023-01-11T22:34:57.3995110Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.3995649Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.3996224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.3996753Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.3997337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.3997787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.3998346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.3998817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.3999276Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.3999763Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4000431Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4001136Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4001666Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4002123Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4002482Z dist init r=1, world=2
2023-01-11T22:34:57.4002733Z dist init r=0, world=2
2023-01-11T22:34:57.4002954Z ok (3.811s)
2023-01-11T22:34:57.4003375Z   test_default_communication_hook_behavior_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2023-01-11T22:34:57.4004136Z Tests FSDP's default communication hook's behavior and correctness. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70815
2023-01-11T22:34:57.4005022Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70816
2023-01-11T22:34:57.4005623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4006078Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4006660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4007135Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4007699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4008148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4008722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4009180Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4009637Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4010136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4010802Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4011478Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4012004Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4012481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4012841Z dist init r=1, world=2
2023-01-11T22:34:57.4013076Z dist init r=0, world=2
2023-01-11T22:34:57.4013415Z ok (3.811s)
2023-01-11T22:34:57.4013883Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:34:57.4014711Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70898
2023-01-11T22:34:57.4015239Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70899
2023-01-11T22:34:57.4015857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4016312Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4016874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4017348Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4017932Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4018361Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4018943Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4019407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4019865Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4020344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4021323Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4022069Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4022844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4023307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4023914Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4024300Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4024819Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4025211Z     _check_comm_hook(
2023-01-11T22:34:57.4025724Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:34:57.4026206Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:34:57.4026756Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4027149Z     traceback.print_stack()
2023-01-11T22:34:57.4027659Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4028028Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4028558Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4028948Z     _check_comm_hook(
2023-01-11T22:34:57.4029441Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:34:57.4029916Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:34:57.4030479Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4030859Z     traceback.print_stack()
2023-01-11T22:34:57.4031342Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4031805Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4032346Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4032775Z     _check_comm_hook(
2023-01-11T22:34:57.4033288Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:34:57.4033661Z     p_assert(
2023-01-11T22:34:57.4034133Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4034493Z     traceback.print_stack()
2023-01-11T22:34:57.4034990Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4035369Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4035880Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4036272Z     _check_comm_hook(
2023-01-11T22:34:57.4036782Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:34:57.4037160Z     p_assert(
2023-01-11T22:34:57.4037610Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4037994Z     traceback.print_stack()
2023-01-11T22:34:57.4038263Z dist init r=1, world=2
2023-01-11T22:34:57.4038533Z Communication hook should not be `None`
2023-01-11T22:34:57.4038864Z Communication hook state should not be `None`
2023-01-11T22:34:57.4039156Z dist init r=0, world=2
2023-01-11T22:34:57.4039422Z Communication hook should not be `None`
2023-01-11T22:34:57.4039747Z Communication hook state should not be `None`
2023-01-11T22:34:57.4040025Z ok (3.711s)
2023-01-11T22:34:57.4040464Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:34:57.4041222Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 70981
2023-01-11T22:34:57.4041750Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 70982
2023-01-11T22:34:57.4042370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4042803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4043383Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4043857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4044860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4045330Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4045917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4046386Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4046831Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4047331Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4064404Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4065249Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4065794Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4066262Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4067033Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4067453Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4068058Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4068460Z     _check_comm_hook(
2023-01-11T22:34:57.4068983Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:34:57.4069464Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:34:57.4070012Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4070398Z     traceback.print_stack()
2023-01-11T22:34:57.4070902Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4071279Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4071814Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4072209Z     _check_comm_hook(
2023-01-11T22:34:57.4072724Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:34:57.4073182Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:34:57.4073746Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4074132Z     traceback.print_stack()
2023-01-11T22:34:57.4074615Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4074999Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4075533Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4075925Z     _check_comm_hook(
2023-01-11T22:34:57.4076426Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:34:57.4076808Z     p_assert(
2023-01-11T22:34:57.4077285Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4077648Z     traceback.print_stack()
2023-01-11T22:34:57.4078145Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4078529Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4079042Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4079431Z     _check_comm_hook(
2023-01-11T22:34:57.4079950Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:34:57.4080324Z     p_assert(
2023-01-11T22:34:57.4080781Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4081161Z     traceback.print_stack()
2023-01-11T22:34:57.4081438Z dist init r=0, world=2
2023-01-11T22:34:57.4081705Z Communication hook should not be `None`
2023-01-11T22:34:57.4082035Z Communication hook state should not be `None`
2023-01-11T22:34:57.4082327Z dist init r=1, world=2
2023-01-11T22:34:57.4082591Z Communication hook should not be `None`
2023-01-11T22:34:57.4082917Z Communication hook state should not be `None`
2023-01-11T22:34:57.4083199Z ok (3.811s)
2023-01-11T22:34:57.4083646Z   test_default_communication_hook_initialization_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2023-01-11T22:34:57.4084805Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71064
2023-01-11T22:34:57.4085442Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71065
2023-01-11T22:34:57.4086088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4086605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4087193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4087671Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4088256Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4088691Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4089264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4089738Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4090184Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4090691Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4091362Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4092062Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4092573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4093053Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4093656Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4094045Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4094568Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4094960Z     _check_comm_hook(
2023-01-11T22:34:57.4095479Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:34:57.4095939Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:34:57.4096498Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4096885Z     traceback.print_stack()
2023-01-11T22:34:57.4097367Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4097749Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4098282Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4098676Z     _check_comm_hook(
2023-01-11T22:34:57.4099169Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:34:57.4099647Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:34:57.4100207Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4100573Z     traceback.print_stack()
2023-01-11T22:34:57.4101072Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4101455Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4101986Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4102357Z     _check_comm_hook(
2023-01-11T22:34:57.4102868Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:34:57.4103305Z     p_assert(
2023-01-11T22:34:57.4103775Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4104228Z     traceback.print_stack()
2023-01-11T22:34:57.4104732Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4105115Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4105631Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4106020Z     _check_comm_hook(
2023-01-11T22:34:57.4106529Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:34:57.4106886Z     p_assert(
2023-01-11T22:34:57.4107355Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4107733Z     traceback.print_stack()
2023-01-11T22:34:57.4107985Z dist init r=1, world=2
2023-01-11T22:34:57.4108272Z Communication hook should not be `None`
2023-01-11T22:34:57.4108599Z Communication hook state should not be `None`
2023-01-11T22:34:57.4108879Z dist init r=0, world=2
2023-01-11T22:34:57.4109158Z Communication hook should not be `None`
2023-01-11T22:34:57.4109486Z Communication hook state should not be `None`
2023-01-11T22:34:57.4109767Z ok (3.812s)
2023-01-11T22:34:57.4110201Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:34:57.4110958Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71147
2023-01-11T22:34:57.4111486Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71148
2023-01-11T22:34:57.4112082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4112543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4113123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4113600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4114168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4114621Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4115198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4115669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4116111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4116624Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4117290Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4117972Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4118499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4118976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4119571Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4119944Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4120483Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4120875Z     _check_comm_hook(
2023-01-11T22:34:57.4121431Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:34:57.4121923Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:34:57.4122540Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4122929Z     traceback.print_stack()
2023-01-11T22:34:57.4123415Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4123803Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4124680Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4125064Z     _check_comm_hook(
2023-01-11T22:34:57.4125584Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:34:57.4126069Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:34:57.4126633Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4127007Z     traceback.print_stack()
2023-01-11T22:34:57.4127503Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4127884Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4128398Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4128790Z     _check_comm_hook(
2023-01-11T22:34:57.4129298Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:34:57.4129672Z     p_assert(
2023-01-11T22:34:57.4130127Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4130515Z     traceback.print_stack()
2023-01-11T22:34:57.4131018Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4131388Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4131917Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4132307Z     _check_comm_hook(
2023-01-11T22:34:57.4132801Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:34:57.4133178Z     p_assert(
2023-01-11T22:34:57.4133648Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4134029Z     traceback.print_stack()
2023-01-11T22:34:57.4134278Z dist init r=0, world=2
2023-01-11T22:34:57.4134563Z Communication hook should not be `None`
2023-01-11T22:34:57.4134891Z Communication hook state should not be `None`
2023-01-11T22:34:57.4135167Z dist init r=1, world=2
2023-01-11T22:34:57.4135449Z Communication hook should not be `None`
2023-01-11T22:34:57.4135777Z Communication hook state should not be `None`
2023-01-11T22:34:57.4136043Z ok (3.811s)
2023-01-11T22:34:57.4136495Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:34:57.4137253Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71230
2023-01-11T22:34:57.4137784Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71231
2023-01-11T22:34:57.4138377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4138832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4139500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4139975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4140569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4141095Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4141672Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4142125Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4142585Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4143095Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4143743Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4144446Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4144979Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4145501Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4146086Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4146473Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4146970Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4147345Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4147862Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4148253Z     _check_comm_hook(
2023-01-11T22:34:57.4148765Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:34:57.4149223Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:34:57.4149826Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4150220Z     _check_comm_hook(
2023-01-11T22:34:57.4150702Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4151070Z     traceback.print_stack()
2023-01-11T22:34:57.4151594Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:34:57.4152067Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:34:57.4152609Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4152991Z     traceback.print_stack()
2023-01-11T22:34:57.4153491Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4153877Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4154388Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4154776Z     _check_comm_hook(
2023-01-11T22:34:57.4155287Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:34:57.4155645Z     p_assert(
2023-01-11T22:34:57.4156111Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4156493Z     traceback.print_stack()
2023-01-11T22:34:57.4156974Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4157418Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4157967Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4158423Z     _check_comm_hook(
2023-01-11T22:34:57.4159055Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:34:57.4159433Z     p_assert(
2023-01-11T22:34:57.4159905Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4160271Z     traceback.print_stack()
2023-01-11T22:34:57.4160542Z dist init r=1, world=2
2023-01-11T22:34:57.4160828Z Communication hook should not be `None`
2023-01-11T22:34:57.4161140Z Communication hook state should not be `None`
2023-01-11T22:34:57.4161430Z dist init r=0, world=2
2023-01-11T22:34:57.4161710Z Communication hook should not be `None`
2023-01-11T22:34:57.4162040Z Communication hook state should not be `None`
2023-01-11T22:34:57.4162305Z ok (3.812s)
2023-01-11T22:34:57.4162764Z   test_default_communication_hook_initialization_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2023-01-11T22:34:57.4163533Z Tests FSDP's communication hook interface behavior. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71313
2023-01-11T22:34:57.4164043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71314
2023-01-11T22:34:57.4165041Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4165502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4166085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4166545Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4167136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4167585Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4168147Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4168622Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4169082Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4169590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4170233Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4170927Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4171456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4171933Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4172516Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4172904Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4173409Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4173774Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4174305Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4174693Z     _check_comm_hook(
2023-01-11T22:34:57.4175209Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4175575Z     _check_comm_hook(
2023-01-11T22:34:57.4176199Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:34:57.4176692Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:34:57.4177347Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 821, in _check_comm_hook
2023-01-11T22:34:57.4177820Z     p_assert(comm_hook is not None, "Communication hook should not be `None`")
2023-01-11T22:34:57.4178381Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4178762Z     traceback.print_stack()
2023-01-11T22:34:57.4179237Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4179619Z     traceback.print_stack()
2023-01-11T22:34:57.4180120Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4180490Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4181026Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4181420Z     _check_comm_hook(
2023-01-11T22:34:57.4181917Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:34:57.4182295Z     p_assert(
2023-01-11T22:34:57.4182768Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4183152Z     traceback.print_stack()
2023-01-11T22:34:57.4183630Z   File "/opt/conda/lib/python3.10/site-packages/torch/autograd/grad_mode.py", line 34, in decorate_context
2023-01-11T22:34:57.4184006Z     return func(*args, **kwargs)
2023-01-11T22:34:57.4184538Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 674, in _post_backward_hook
2023-01-11T22:34:57.4184912Z     _check_comm_hook(
2023-01-11T22:34:57.4185416Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_runtime_utils.py", line 822, in _check_comm_hook
2023-01-11T22:34:57.4185796Z     p_assert(
2023-01-11T22:34:57.4186267Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_utils.py", line 116, in p_assert
2023-01-11T22:34:57.4186632Z     traceback.print_stack()
2023-01-11T22:34:57.4186901Z dist init r=1, world=2
2023-01-11T22:34:57.4187185Z Communication hook should not be `None`
2023-01-11T22:34:57.4187497Z Communication hook state should not be `None`
2023-01-11T22:34:57.4187785Z dist init r=0, world=2
2023-01-11T22:34:57.4188066Z Communication hook should not be `None`
2023-01-11T22:34:57.4188369Z Communication hook state should not be `None`
2023-01-11T22:34:57.4188645Z ok (3.811s)
2023-01-11T22:34:57.4189171Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71396
2023-01-11T22:34:57.4189761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71397
2023-01-11T22:34:57.4190357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4190797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4191379Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4191833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4192408Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4192843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4193417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4193926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4194399Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4194964Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4195616Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4196319Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4196848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4197323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4197665Z dist init r=1, world=2
2023-01-11T22:34:57.4197922Z dist init r=0, world=2
2023-01-11T22:34:57.4198168Z ok (3.810s)
2023-01-11T22:34:57.4198671Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71479
2023-01-11T22:34:57.4199288Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71480
2023-01-11T22:34:57.4199910Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4200367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4200933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4201407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4201995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4202451Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4203019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4203496Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4203951Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4204733Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4205605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4206307Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4206838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4207306Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4207665Z dist init r=1, world=2
2023-01-11T22:34:57.4207921Z dist init r=0, world=2
2023-01-11T22:34:57.4208151Z ok (3.710s)
2023-01-11T22:34:57.4208670Z   test_fp16_hook_has_wrapping_False_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71562
2023-01-11T22:34:57.4209281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71563
2023-01-11T22:34:57.4209959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4210398Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4210975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4211538Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4212127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4212658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4213241Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4213713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4214152Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4214657Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4215321Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4216021Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4216531Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4217005Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4217365Z dist init r=0, world=2
2023-01-11T22:34:57.4217599Z dist init r=1, world=2
2023-01-11T22:34:57.4217840Z ok (3.910s)
2023-01-11T22:34:57.4218352Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71645
2023-01-11T22:34:57.4218958Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71646
2023-01-11T22:34:57.4219562Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4220023Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4220601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4221063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4221642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4222088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4222652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4223099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4223557Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4224064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4224735Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4225417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4225951Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4226431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4226774Z dist init r=0, world=2
2023-01-11T22:34:57.4227030Z dist init r=1, world=2
2023-01-11T22:34:57.4227269Z ok (3.810s)
2023-01-11T22:34:57.4227763Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71728
2023-01-11T22:34:57.4228372Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71729
2023-01-11T22:34:57.4229061Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4229527Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4230154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4230633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4231215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4231663Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4232220Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4232687Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4233146Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4233632Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4234300Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4234998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4235527Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4235988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4236348Z dist init r=0, world=2
2023-01-11T22:34:57.4236600Z dist init r=1, world=2
2023-01-11T22:34:57.4236822Z ok (3.810s)
2023-01-11T22:34:57.4237349Z   test_fp16_hook_has_wrapping_True_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71811
2023-01-11T22:34:57.4237963Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71812
2023-01-11T22:34:57.4238585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4239021Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4239593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4240068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4240648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4241077Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4241649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4242115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4242559Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4243064Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4243724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4244623Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4245141Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4245661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4246023Z dist init r=1, world=2
2023-01-11T22:34:57.4246350Z dist init r=0, world=2
2023-01-11T22:34:57.4246610Z ok (3.810s)
2023-01-11T22:34:57.4247012Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:34:57.4247818Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71894
2023-01-11T22:34:57.4248342Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71895
2023-01-11T22:34:57.4248957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4249412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4249994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4250454Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4251045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4251497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4252060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4252531Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4252988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4253495Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4254145Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4254846Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4255376Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4255855Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4256202Z dist init r=1, world=2
2023-01-11T22:34:57.4256454Z dist init r=0, world=2
2023-01-11T22:34:57.4256696Z ok (3.410s)
2023-01-11T22:34:57.4257076Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:34:57.4257792Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 71973
2023-01-11T22:34:57.4258329Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 71974
2023-01-11T22:34:57.4258922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4259384Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4259968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4260449Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4261016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4261467Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4262045Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4262515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4262956Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4263462Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4264193Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4264888Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4265477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4265955Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4266312Z dist init r=0, world=2
2023-01-11T22:34:57.4266548Z dist init r=1, world=2
2023-01-11T22:34:57.4266792Z ok (3.310s)
2023-01-11T22:34:57.4267204Z   test_registering_hook_non_root_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2023-01-11T22:34:57.4267914Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72052
2023-01-11T22:34:57.4268465Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72053
2023-01-11T22:34:57.4269077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4269538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4270094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4270565Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4271148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4271576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4272150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4272621Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4273078Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4273568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4274232Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4274931Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4275459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4275910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4276260Z dist init r=0, world=2
2023-01-11T22:34:57.4276513Z dist init r=1, world=2
2023-01-11T22:34:57.4276737Z ok (3.310s)
2023-01-11T22:34:57.4277146Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_FULL_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:34:57.4277877Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72131
2023-01-11T22:34:57.4278419Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72132
2023-01-11T22:34:57.4279016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4279472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4280053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4280513Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4281155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4281616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4282196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4282721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4283178Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4283687Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4284576Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4285293Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4285826Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4286304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4286645Z dist init r=1, world=2
2023-01-11T22:34:57.4286902Z dist init r=0, world=2
2023-01-11T22:34:57.4287149Z ok (3.209s)
2023-01-11T22:34:57.4287535Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_NO_SHARD (__main__.TestCommunicationHooks)
2023-01-11T22:34:57.4288256Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72210
2023-01-11T22:34:57.4288798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72211
2023-01-11T22:34:57.4289416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4289855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4290427Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4290876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4291463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4291919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4292513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4292983Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4293425Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4293931Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4294601Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4295296Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4295811Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4296289Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4296648Z dist init r=1, world=2
2023-01-11T22:34:57.4296883Z dist init r=0, world=2
2023-01-11T22:34:57.4297124Z ok (3.310s)
2023-01-11T22:34:57.4297539Z   test_registering_hook_submodules_sharding_strategy_ShardingStrategy_SHARD_GRAD_OP (__main__.TestCommunicationHooks)
2023-01-11T22:34:57.4298268Z Tests FSDP's communication hook registering for submodules. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72289
2023-01-11T22:34:57.4298788Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 72290
2023-01-11T22:34:57.4299486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4299960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4300593Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4301072Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4301654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:34:57.4302105Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:34:57.4302660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:34:57.4303132Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:34:57.4303596Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:34:57.4304105Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:34:57.4304761Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4305463Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:34:57.4305992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:34:57.4306452Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:34:57.4306815Z dist init r=1, world=2
2023-01-11T22:34:57.4307069Z dist init r=0, world=2
2023-01-11T22:34:57.4307290Z ok (3.310s)
2023-01-11T22:34:57.4307442Z 
2023-01-11T22:34:57.4307715Z ----------------------------------------------------------------------
2023-01-11T22:34:57.4308058Z Ran 27 tests in 101.296s
2023-01-11T22:34:57.4308227Z 
2023-01-11T22:34:57.4308320Z OK
2023-01-11T22:34:57.4308438Z 
2023-01-11T22:34:57.4308563Z Generating XML reports...
2023-01-11T22:34:57.4309196Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20230111223315.xml
2023-01-11T22:34:57.4309573Z 
2023-01-11T22:34:57.4309931Z ##[endgroup]
2023-01-11T22:34:57.4310532Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_comm_hooks (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_comm_hooks_tcezadhs)
2023-01-11T22:34:57.4310901Z 
2023-01-11T22:34:57.4311159Z Running distributed/test_c10d_pypg ... [2023-01-11 22:34:57.390588]
2023-01-11T22:34:57.4311818Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/test_c10d_pypg.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:34:57.390838]
2023-01-11T22:37:02.0953230Z 
2023-01-11T22:37:02.0954020Z Expand the folded group to see the log file of distributed/test_c10d_pypg
2023-01-11T22:37:02.0955017Z ##[group]PRINTING LOG FILE of distributed/test_c10d_pypg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_pypg_43lzg27o)
2023-01-11T22:37:02.0955368Z 
2023-01-11T22:37:02.0957640Z Running tests...
2023-01-11T22:37:02.0958460Z ----------------------------------------------------------------------
2023-01-11T22:37:02.0959516Z Test results will be stored in test-reports/python-unittest/distributed.test_c10d_pypg
2023-01-11T22:37:02.0959987Z   test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkSubclass)
2023-01-11T22:37:02.0960614Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:37:02.0961124Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72403
2023-01-11T22:37:02.0961755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.0962459Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.0963086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.0963682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.0964117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.0965124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzv53lbcc
2023-01-11T22:37:02.0966049Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzv53lbcc/_remote_module_non_scriptable.py
2023-01-11T22:37:02.0966720Z ok (5.059s)
2023-01-11T22:37:02.0967487Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkSubclass)
2023-01-11T22:37:02.0968526Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72440
2023-01-11T22:37:02.0969276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.0969744Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.0971895Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.0972807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.0973647Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.0974508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_7ws3qqy
2023-01-11T22:37:02.0975458Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_7ws3qqy/_remote_module_non_scriptable.py
2023-01-11T22:37:02.0976077Z ok (3.408s)
2023-01-11T22:37:02.0976679Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2023-01-11T22:37:02.0977632Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72477
2023-01-11T22:37:02.0978883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.0979759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.0980797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.0981280Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.0981706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.0982216Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmq6brxe_
2023-01-11T22:37:02.0982770Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmq6brxe_/_remote_module_non_scriptable.py
2023-01-11T22:37:02.0983295Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.0983768Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.0984955Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:37:02.0985746Z   warnings.warn(
2023-01-11T22:37:02.0986128Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.0986618Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.0987247Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.0987748Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.0988167Z ok (3.609s)
2023-01-11T22:37:02.0988510Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2023-01-11T22:37:02.0989073Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72514
2023-01-11T22:37:02.0989780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.0990238Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.0990803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.0991279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.0991733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.0992227Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoldpr6dk
2023-01-11T22:37:02.0992782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoldpr6dk/_remote_module_non_scriptable.py
2023-01-11T22:37:02.0993306Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.0993801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.0994960Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:37:02.0995697Z   warnings.warn(
2023-01-11T22:37:02.0996080Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.0996567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.0997036Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.0997518Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.0997866Z ok (3.609s)
2023-01-11T22:37:02.0998254Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2023-01-11T22:37:02.0998952Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72551
2023-01-11T22:37:02.0999656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1000119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1000686Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1001168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1001615Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1002125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp42vqfc46
2023-01-11T22:37:02.1002652Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp42vqfc46/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1003172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1003661Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1003997Z ok (3.508s)
2023-01-11T22:37:02.1005168Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2023-01-11T22:37:02.1005906Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72588
2023-01-11T22:37:02.1006674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1007111Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1007694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1008167Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1008612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1009101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0p848a80
2023-01-11T22:37:02.1009646Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0p848a80/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1010165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1010637Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1010999Z ok (3.508s)
2023-01-11T22:37:02.1011367Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2023-01-11T22:37:02.1012092Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72625
2023-01-11T22:37:02.1012788Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1013242Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1013825Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1014308Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1014733Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1015246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk38ax7d_
2023-01-11T22:37:02.1015795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk38ax7d_/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1016297Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1017348Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:37:02.1018372Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1018728Z ok (3.509s)
2023-01-11T22:37:02.1019089Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2023-01-11T22:37:02.1019795Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72662
2023-01-11T22:37:02.1020502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1020955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1021519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1022050Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1022504Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1023008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprfuzop2b
2023-01-11T22:37:02.1023587Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprfuzop2b/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1023971Z ok (3.408s)
2023-01-11T22:37:02.1024326Z   test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkSubclass)
2023-01-11T22:37:02.1024877Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72699
2023-01-11T22:37:02.1025591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1026043Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1026619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1027074Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1027520Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1028023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2t2md0pq
2023-01-11T22:37:02.1028566Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2t2md0pq/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1029073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1029559Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1029910Z ok (3.508s)
2023-01-11T22:37:02.1030267Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2023-01-11T22:37:02.1030852Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72736
2023-01-11T22:37:02.1031560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1032014Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1032571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1033044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1033485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1033989Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnf6z5iui
2023-01-11T22:37:02.1034515Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnf6z5iui/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1035587Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:37:02.1037278Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:37:02.1038009Z   warnings.warn(
2023-01-11T22:37:02.1038441Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1038925Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1039326Z ok (3.509s)
2023-01-11T22:37:02.1039698Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2023-01-11T22:37:02.1040265Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72773
2023-01-11T22:37:02.1040979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1041431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1042004Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1042464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1042911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1043420Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy4qbqn3o
2023-01-11T22:37:02.1043968Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy4qbqn3o/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1045579Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:37:02.1046315Z   warnings.warn(
2023-01-11T22:37:02.1046691Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1047184Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1047519Z ok (3.508s)
2023-01-11T22:37:02.1047894Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkSubclass)
2023-01-11T22:37:02.1048457Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72810
2023-01-11T22:37:02.1049120Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1049578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1050155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1050631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1051098Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1051607Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpje6et9lc
2023-01-11T22:37:02.1052152Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpje6et9lc/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1052676Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1053143Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1053494Z ok (3.408s)
2023-01-11T22:37:02.1053867Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkSubclass)
2023-01-11T22:37:02.1054405Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72847
2023-01-11T22:37:02.1055086Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1055540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1056195Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1056667Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1057169Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1057671Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwrzdqj1i
2023-01-11T22:37:02.1058200Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwrzdqj1i/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1058720Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1059207Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1059688Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1060155Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1060502Z ok (3.509s)
2023-01-11T22:37:02.1060951Z   test_ddp_invoke_work_object (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72884
2023-01-11T22:37:02.1061642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1062096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1062669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1063142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1063571Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1064073Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg2qwi8tt
2023-01-11T22:37:02.1064621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg2qwi8tt/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1065002Z ok (2.231s)
2023-01-11T22:37:02.1065425Z   test_ddp_with_pypg (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72920
2023-01-11T22:37:02.1066113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1066562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1067122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1067589Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1068030Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1068537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiiawcpfv
2023-01-11T22:37:02.1069068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiiawcpfv/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1069592Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1069944Z ok (2.206s)
2023-01-11T22:37:02.1070378Z   test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72956
2023-01-11T22:37:02.1071094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1071546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1072118Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1072575Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1073069Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1073581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaxk9j5n8
2023-01-11T22:37:02.1074150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaxk9j5n8/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1074665Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1075017Z ok (2.206s)
2023-01-11T22:37:02.1075466Z   test_invalid_powerSGD_state (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 72992
2023-01-11T22:37:02.1076156Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1076607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1077193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1077666Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1078087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1078891Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:37:02.1079988Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:37:02.1081072Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:37:02.1082152Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:37:02.1083231Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:37:02.1084609Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:37:02.1085238Z ok (2.206s)
2023-01-11T22:37:02.1085690Z   test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73026
2023-01-11T22:37:02.1086399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1086852Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1087507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1087992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1088432Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1088980Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprmjv1d4m
2023-01-11T22:37:02.1089524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprmjv1d4m/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1090043Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1090399Z ok (2.908s)
2023-01-11T22:37:02.1090838Z   test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkSubclass) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73063
2023-01-11T22:37:02.1091557Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1092016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1092578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1093054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1093499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1094003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkdwszizc
2023-01-11T22:37:02.1094532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkdwszizc/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1095054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1095408Z ok (3.108s)
2023-01-11T22:37:02.1095736Z   test_ddp_checkpointing_dynamic_module (__main__.TestDDPWithWorkWrapper)
2023-01-11T22:37:02.1096442Z Dynamic module can be checkpointed, multiple times, with non-reentrant ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73100
2023-01-11T22:37:02.1097148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1097602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1098163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1098641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1099085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1099588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq_tld9m0
2023-01-11T22:37:02.1100112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq_tld9m0/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1100495Z ok (3.509s)
2023-01-11T22:37:02.1100856Z   test_ddp_checkpointing_dynamic_weight_sharing (__main__.TestDDPWithWorkWrapper)
2023-01-11T22:37:02.1101411Z Dynamic module can be checkpointed multiple times with weight sharing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73137
2023-01-11T22:37:02.1102121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1102572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1103150Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1103608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1104049Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1104556Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwvag3jyl
2023-01-11T22:37:02.1105156Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwvag3jyl/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1105534Z ok (3.408s)
2023-01-11T22:37:02.1105891Z   test_ddp_checkpointing_once_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2023-01-11T22:37:02.1106496Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73174
2023-01-11T22:37:02.1107176Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1107624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1108201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1108673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1109102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1109602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoi7_d_vx
2023-01-11T22:37:02.1110141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoi7_d_vx/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1110645Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1111131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1112306Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:37:02.1113037Z   warnings.warn(
2023-01-11T22:37:02.1113398Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1113880Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1114368Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1114850Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1115179Z ok (3.709s)
2023-01-11T22:37:02.1115534Z   test_ddp_checkpointing_once_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2023-01-11T22:37:02.1116094Z DDP works as expected when layer is checkpointed only once. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73211
2023-01-11T22:37:02.1116770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1117219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1117798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1118269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1118694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1119193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1_mur92q
2023-01-11T22:37:02.1119729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1_mur92q/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1120246Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1120714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1121938Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:37:02.1122708Z   warnings.warn(
2023-01-11T22:37:02.1123081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1123543Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1124018Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1124796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1125127Z ok (3.609s)
2023-01-11T22:37:02.1125506Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2023-01-11T22:37:02.1126229Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73248
2023-01-11T22:37:02.1126929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1127371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1127947Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1128420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1128859Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1129341Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbyv1ca6w
2023-01-11T22:37:02.1129880Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbyv1ca6w/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1130395Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1130863Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1131209Z ok (3.508s)
2023-01-11T22:37:02.1131588Z   test_ddp_checkpointing_twice_static_graph_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2023-01-11T22:37:02.1132302Z Regardless of reentrant or non-reentrant checkpointing impl, ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73285
2023-01-11T22:37:02.1132980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1133431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1134007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1134461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1134904Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1135406Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps99zilc9
2023-01-11T22:37:02.1135949Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps99zilc9/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1136450Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1136934Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1137289Z ok (3.509s)
2023-01-11T22:37:02.1137630Z   test_ddp_checkpointing_twice_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2023-01-11T22:37:02.1138347Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73322
2023-01-11T22:37:02.1139058Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1139591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1140167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1140705Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1141147Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1141651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ko7curl
2023-01-11T22:37:02.1142179Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ko7curl/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1142697Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1143741Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:37:02.1144731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1145067Z ok (3.509s)
2023-01-11T22:37:02.1145424Z   test_ddp_checkpointing_twice_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2023-01-11T22:37:02.1146146Z Checkpoitning twice fails for non-static graph with reentrant checkpoint ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73359
2023-01-11T22:37:02.1146863Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1147302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1147884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1148358Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1148782Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1149286Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzvw3h1om
2023-01-11T22:37:02.1149830Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzvw3h1om/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1150214Z ok (3.509s)
2023-01-11T22:37:02.1150550Z   test_ddp_checkpointing_twice_weight_sharing (__main__.TestDDPWithWorkWrapper)
2023-01-11T22:37:02.1151160Z Checkpointing should work with static graph in the case of checkpointing ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73396
2023-01-11T22:37:02.1151870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1152323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1152886Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1153361Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1153801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1154291Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpehqi0j9_
2023-01-11T22:37:02.1154834Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpehqi0j9_/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1155351Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1155896Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1156239Z ok (3.509s)
2023-01-11T22:37:02.1156611Z   test_ddp_checkpointing_unused_params_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2023-01-11T22:37:02.1157250Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73433
2023-01-11T22:37:02.1157940Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1158394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1158969Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1159437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1159862Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1160366Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj6mt65fy
2023-01-11T22:37:02.1160907Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj6mt65fy/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1161980Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:37:02.1163665Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:37:02.1164767Z   warnings.warn(
2023-01-11T22:37:02.1165145Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1165638Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1165990Z ok (3.509s)
2023-01-11T22:37:02.1166342Z   test_ddp_checkpointing_unused_params_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2023-01-11T22:37:02.1166926Z With reentrant autograd checkpointing impl, DDP will fail when there are ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73470
2023-01-11T22:37:02.1167641Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1168097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1168664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1169140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1169580Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1170068Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2io09mpj
2023-01-11T22:37:02.1170613Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2io09mpj/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1171892Z /opt/conda/lib/python3.10/site-packages/torch/nn/parallel/distributed.py:1911: UserWarning: You passed find_unused_parameters=true to DistributedDataParallel, `_set_static_graph` will detect unused parameters automatically, so you do not need to set find_unused_parameters=true, just be sure these unused parameters will not change during training loop while calling `_set_static_graph`.
2023-01-11T22:37:02.1172639Z   warnings.warn(
2023-01-11T22:37:02.1173000Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1173549Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1173901Z ok (3.509s)
2023-01-11T22:37:02.1174276Z   test_ddp_checkpointing_weight_sharing_use_reentrant_False (__main__.TestDDPWithWorkWrapper)
2023-01-11T22:37:02.1174814Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73507
2023-01-11T22:37:02.1175500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1175952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1176514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1176993Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1177436Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1177944Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsoctsnlh
2023-01-11T22:37:02.1178468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsoctsnlh/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1178994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1179482Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1179816Z ok (3.509s)
2023-01-11T22:37:02.1180186Z   test_ddp_checkpointing_weight_sharing_use_reentrant_True (__main__.TestDDPWithWorkWrapper)
2023-01-11T22:37:02.1180742Z Test that checkpointing with weight sharing works. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73544
2023-01-11T22:37:02.1181425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1181860Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1182440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1182912Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1183351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1183839Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1295kel5
2023-01-11T22:37:02.1184377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1295kel5/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1184890Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1185361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1185847Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1186336Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1186681Z ok (3.509s)
2023-01-11T22:37:02.1187110Z   test_ddp_invoke_work_object (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73581
2023-01-11T22:37:02.1187814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1188269Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1188826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1189300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1189796Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1190309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0t93sxt6
2023-01-11T22:37:02.1190882Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0t93sxt6/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1191263Z ok (2.207s)
2023-01-11T22:37:02.1191700Z   test_ddp_with_pypg (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73617
2023-01-11T22:37:02.1192399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1192831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1193407Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1193880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1194303Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1194813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpazz7t1tm
2023-01-11T22:37:02.1195354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpazz7t1tm/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1195873Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1196209Z ok (2.206s)
2023-01-11T22:37:02.1196661Z   test_ddp_with_pypg_with_grad_views (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73653
2023-01-11T22:37:02.1197367Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1197797Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1198377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1198848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1199290Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1199775Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2j3iimzk
2023-01-11T22:37:02.1200315Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2j3iimzk/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1200833Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1201167Z ok (2.106s)
2023-01-11T22:37:02.1201616Z   test_invalid_powerSGD_state (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73689
2023-01-11T22:37:02.1202325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1202777Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1203334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1203804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1204408Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1205220Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:37:02.1206396Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:37:02.1207544Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 0; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:37:02.1208628Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:37:02.1209704Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = True; warm_start = False; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:37:02.1210782Z INFO:torch.distributed.algorithms.ddp_comm_hooks.powerSGD_hook:PowerSGD config: matrix_approximation_rank = 1; start_powerSGD_iter = 1; min_compression_rate = 2; orthogonalization_epsilon = 0; use_error_feedback = False; warm_start = True; random_seed = 0; compression_stats_logging_frequency = 10000; batch_tensors_with_same_shape = False
2023-01-11T22:37:02.1211414Z ok (2.106s)
2023-01-11T22:37:02.1211869Z   test_sync_batch_norm_empty_input (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73723
2023-01-11T22:37:02.1212578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1213032Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1213609Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1214088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1214515Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1215024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoco1b6u0
2023-01-11T22:37:02.1215570Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoco1b6u0/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1216073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1216425Z ok (3.009s)
2023-01-11T22:37:02.1216887Z   test_sync_batch_norm_only_empty_input (__main__.TestDDPWithWorkWrapper) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73760
2023-01-11T22:37:02.1217599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:37:02.1218038Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:37:02.1218613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:37:02.1219087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:37:02.1219508Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:37:02.1220012Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7bj3f6f9
2023-01-11T22:37:02.1220552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7bj3f6f9/_remote_module_non_scriptable.py
2023-01-11T22:37:02.1221072Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:37:02.1221462Z ok (3.008s)
2023-01-11T22:37:02.1221622Z 
2023-01-11T22:37:02.1221898Z ----------------------------------------------------------------------
2023-01-11T22:37:02.1222276Z Ran 38 tests in 122.386s
2023-01-11T22:37:02.1222443Z 
2023-01-11T22:37:02.1222518Z OK
2023-01-11T22:37:02.1222653Z 
2023-01-11T22:37:02.1222774Z Generating XML reports...
2023-01-11T22:37:02.1223381Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkSubclass-20230111223459.xml
2023-01-11T22:37:02.1224162Z Generated XML report: test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkWrapper-20230111223459.xml
2023-01-11T22:37:02.1224515Z 
2023-01-11T22:37:02.1224904Z ##[endgroup]
2023-01-11T22:37:02.1225471Z FINISHED PRINTING LOG FILE of distributed/test_c10d_pypg (/var/lib/jenkins/workspace/test/test-reports/distributed-test_c10d_pypg_43lzg27o)
2023-01-11T22:37:02.1225800Z 
2023-01-11T22:37:02.1226090Z Running distributed/fsdp/test_fsdp_use_orig_params ... [2023-01-11 22:37:02.095757]
2023-01-11T22:37:02.1226797Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_use_orig_params.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:37:02.096026]
2023-01-11T22:39:58.4989015Z 
2023-01-11T22:39:58.4991557Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_use_orig_params
2023-01-11T22:39:58.4992580Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_use_orig_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_use_orig_params_4hs5pk_y)
2023-01-11T22:39:58.4993192Z 
2023-01-11T22:39:58.4993313Z Running tests...
2023-01-11T22:39:58.4993855Z ----------------------------------------------------------------------
2023-01-11T22:39:58.4996287Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params
2023-01-11T22:39:58.4997151Z   test_named_parameters_in_forward (__main__.TestFSDPUseOrigParamsFQNs)
2023-01-11T22:39:58.4998067Z Tests that calling ``named_parameters()`` during forward returns FQNs ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T22:39:58.4998585Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73832
2023-01-11T22:39:58.4999057Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73833
2023-01-11T22:39:58.5000016Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5000629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5001709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5002457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5003556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5004634Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5008034Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5008544Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5009032Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5009548Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5010227Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5010907Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5011444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5012270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5012816Z dist init r=1, world=2
2023-01-11T22:39:58.5013075Z dist init r=0, world=2
2023-01-11T22:39:58.5013434Z ok (5.314s)
2023-01-11T22:39:58.5013754Z   test_param_and_buffer_names (__main__.TestFSDPUseOrigParamsFQNs)
2023-01-11T22:39:58.5014296Z Tests that, for ``use_orig_params=True``, the parameter and buffer ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73911
2023-01-11T22:39:58.5015070Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73912
2023-01-11T22:39:58.5015841Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5016282Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5016894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5017376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5017967Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5018409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5018986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5019463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5019925Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5020414Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5021086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5021791Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5022322Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5022792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5024077Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:39:58.5024947Z   warnings.warn(
2023-01-11T22:39:58.5026120Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:39:58.5026913Z   warnings.warn(
2023-01-11T22:39:58.5027167Z dist init r=0, world=2
2023-01-11T22:39:58.5027403Z dist init r=1, world=2
2023-01-11T22:39:58.5027640Z ok (3.409s)
2023-01-11T22:39:58.5028089Z   test_diff_hyperparams_cpu_offload_sharding_strategy_str_full_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2023-01-11T22:39:58.5028715Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 73990
2023-01-11T22:39:58.5029265Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 73991
2023-01-11T22:39:58.5029975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5030444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5031079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5031561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5032155Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5032605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5033473Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5033959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5034431Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5034940Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5035599Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5036310Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5036836Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5037321Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5037787Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5038278Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5038646Z dist init r=0, world=2
2023-01-11T22:39:58.5038891Z dist init r=1, world=2
2023-01-11T22:39:58.5039133Z ok (4.712s)
2023-01-11T22:39:58.5039581Z   test_diff_hyperparams_cpu_offload_sharding_strategy_str_no_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2023-01-11T22:39:58.5040219Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74073
2023-01-11T22:39:58.5040772Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74074
2023-01-11T22:39:58.5041395Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5041851Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5042413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5042889Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5043475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5043924Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5044856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5045330Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5045786Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5046265Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5046930Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5047726Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5048274Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5048730Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5049273Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5049765Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5050129Z dist init r=0, world=2
2023-01-11T22:39:58.5050362Z dist init r=1, world=2
2023-01-11T22:39:58.5050600Z ok (4.613s)
2023-01-11T22:39:58.5051050Z   test_diff_hyperparams_cpu_offload_sharding_strategy_str_shard_grad_op (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2023-01-11T22:39:58.5051686Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74156
2023-01-11T22:39:58.5052235Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74157
2023-01-11T22:39:58.5052857Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5053313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5053875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5054347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5054926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5055355Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5055927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5056394Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5056855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5057333Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5057998Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5058688Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5059263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5059726Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5060205Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5060700Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5061052Z dist init r=1, world=2
2023-01-11T22:39:58.5061302Z dist init r=0, world=2
2023-01-11T22:39:58.5061538Z ok (4.611s)
2023-01-11T22:39:58.5061952Z   test_diff_hyperparams_sharding_strategy_str_full_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2023-01-11T22:39:58.5062586Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74239
2023-01-11T22:39:58.5063132Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74240
2023-01-11T22:39:58.5063747Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5064182Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5064763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5065296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5065890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5066373Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5066946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5067416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5067852Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5068355Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5069017Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5069720Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5070231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5070712Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5071195Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5071682Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5072144Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5072626Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5073098Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5073558Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5074037Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5074506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5074984Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5075449Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5075923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5076396Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5076851Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5077326Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5077799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5078271Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5078724Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5079197Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5079668Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5080119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5080593Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5081062Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5081526Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5082034Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5082506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5083015Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5083466Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5083933Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5084844Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5085316Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5085772Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5086237Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5086714Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5087186Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5087641Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5088108Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5088574Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5089026Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5089492Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5089959Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5090427Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5090882Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5091348Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5091814Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5092263Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5092736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5093202Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5093675Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5094956Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:39:58.5095754Z   warnings.warn(
2023-01-11T22:39:58.5096910Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:39:58.5097689Z   warnings.warn(
2023-01-11T22:39:58.5098061Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5098612Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5099102Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5099635Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5100107Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5100571Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5101041Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5101513Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5101966Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5102437Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5102911Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5103383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5103844Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5104314Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5104783Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5105235Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5105700Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5106165Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5106639Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5107096Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5155736Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5156342Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5156820Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5157308Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5157799Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5158266Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5158738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5159274Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5159758Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5160217Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5160674Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5161131Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5161575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5162034Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5162486Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5162938Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5163537Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5164028Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5165031Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5165493Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5165970Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5166446Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5166919Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5167377Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5167846Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5168322Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5168796Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5169260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5169614Z dist init r=1, world=2
2023-01-11T22:39:58.5169863Z dist init r=0, world=2
2023-01-11T22:39:58.5170083Z ok (32.755s)
2023-01-11T22:39:58.5170497Z   test_diff_hyperparams_sharding_strategy_str_no_shard (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2023-01-11T22:39:58.5171123Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74322
2023-01-11T22:39:58.5171648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74323
2023-01-11T22:39:58.5172308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5172755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5173330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5173783Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5174360Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5174791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5175351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5175796Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5176242Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5176738Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5177382Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5178073Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5178594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5179065Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5179522Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5179995Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5181084Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5182403Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5183642Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5184862Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5186100Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5187321Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5188555Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5189778Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5191001Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5192227Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5192952Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5193417Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5193890Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5194361Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5195348Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5196616Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5197900Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5199124Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5200348Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5201560Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5202787Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5204015Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5205506Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5206709Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5207929Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5209144Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5210360Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5211639Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5212917Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5214128Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5215337Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5216546Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5217752Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5218962Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5219689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5220166Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5220633Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5221091Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5222077Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5223296Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5224526Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5225799Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5227038Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5228303Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5229520Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5230733Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5231955Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5233168Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5234391Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5235611Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5236829Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5238044Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5239258Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5240528Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5241743Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5243005Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5244445Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5245673Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5246403Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5246872Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5247347Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5247821Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5248803Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5250035Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5251246Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5252462Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5253686Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5254892Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5256209Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5257489Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5258712Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5259970Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5261191Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5262403Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5263631Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5264846Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5266061Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5267274Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5268498Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5269724Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5271008Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5272224Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5272998Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5273477Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5273946Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5274405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5275398Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5276624Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5277840Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5279057Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5280287Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5281506Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5282718Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5283932Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5285475Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5286811Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5288112Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5289330Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5290561Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5291774Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5292990Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5294194Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5295424Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5296636Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5297856Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5299085Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5299805Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5300287Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5300747Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5301279Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5302276Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5303561Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5304780Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5305994Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5307287Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5308515Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5309732Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5310954Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5312177Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5313392Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5314605Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5315865Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5317106Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5318374Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5319603Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5320808Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5322029Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5323241Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5324628Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5325863Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5326572Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5327059Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5327533Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5327997Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5328982Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5330206Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5331511Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5332745Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5334040Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5335261Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5336491Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5337708Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5338932Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5340154Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5341369Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5342582Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5343799Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5345017Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5346291Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5347520Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5348783Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5350004Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5351222Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5352434Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5353157Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5353631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5354097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5354575Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5355558Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5356777Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5358005Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5359259Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5360494Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5361779Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5363069Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5364434Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5365671Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5366893Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5367615Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5368081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5368550Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5369022Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5370004Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5371220Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5372450Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5373677Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5374901Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5376112Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5377408Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5378685Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5379908Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5381115Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5382337Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5383540Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5384761Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5385975Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5387193Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5388411Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5389633Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5390848Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5392103Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5393374Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5394093Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5394572Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5395039Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5395506Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5396484Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5397709Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5398937Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5400150Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5401372Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5402580Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5403802Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5405204Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5406504Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5407728Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5409023Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5410243Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5411468Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5412683Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5413900Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5415122Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5416337Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5417549Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5418767Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5419982Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5420699Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5421167Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5421695Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5422184Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5423217Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5424413Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5425641Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5426865Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5428076Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5429281Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5430491Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5431719Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5432937Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5434165Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5435368Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5436643Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5437912Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5439123Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5440343Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5441556Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5442779Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5443993Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5445462Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5446667Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5447393Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5447891Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5448362Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5448827Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5449811Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5451034Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5452335Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5453633Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5454857Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5456075Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5457288Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5458511Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5459784Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5461002Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5462221Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5463438Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5464661Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5465879Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5467168Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5468424Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5469644Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5470864Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5472090Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5473307Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5474033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5474504Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5475790Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:39:58.5476558Z   warnings.warn(
2023-01-11T22:39:58.5477701Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:39:58.5478480Z   warnings.warn(
2023-01-11T22:39:58.5478852Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5479313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5480313Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5481606Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5482861Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5484141Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5485608Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5486828Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5488065Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5489294Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5490516Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5491744Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5492965Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5494184Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5495406Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5496684Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5497923Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5499199Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5500429Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5501640Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5502858Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5504072Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5504794Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5505268Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5505739Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5506214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5507197Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5508417Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5509644Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5510862Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5512142Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5513443Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5514666Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5515890Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5517099Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5518312Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5519527Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5520735Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5521947Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5523162Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5524597Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5525832Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5527132Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5528416Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5529624Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5530840Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5531567Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5532052Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5532525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5532986Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5533974Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5535201Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5536429Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5537645Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5538874Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5540096Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5541305Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5542584Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5543842Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5545047Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5546269Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5547490Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5548713Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5549934Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5551157Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5552374Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5553594Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5554811Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5556036Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5557308Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5558081Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5558549Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5559021Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5559545Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5560542Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5561767Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5562992Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5564409Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5565654Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5566871Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5568099Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5569321Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5570543Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5571843Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5573069Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5574356Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5575581Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5576794Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5578019Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5579236Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5580458Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5581678Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5582900Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5584101Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5584832Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5585313Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5585788Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5586251Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5587287Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5588556Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5589786Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5590998Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5592219Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5593430Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5594643Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5595858Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5597083Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5598299Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5599514Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5600733Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5602009Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5603232Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5604725Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5605949Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5607219Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5608444Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5609666Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5610878Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5611606Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5612090Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5612553Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5613033Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5614019Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5615255Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5616492Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5617785Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5619086Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5620295Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5621520Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5622740Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5623970Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5625194Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5626421Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5627623Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5628852Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5630069Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5631278Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5632551Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5633828Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5635044Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5636266Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5637492Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5638200Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5638683Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5639162Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5639646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5640623Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5641854Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5643084Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5644522Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5645777Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5647078Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5648322Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5649600Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5650826Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5652051Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5653277Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5654500Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5655725Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5656943Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5658165Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5659427Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5660648Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5661872Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5663145Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5664412Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5665145Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5665631Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5666097Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5666577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5667564Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5668782Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5670015Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5671223Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5672462Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5673683Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5674911Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5676132Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5677415Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5678658Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5679942Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5681168Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5682381Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5683611Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5685071Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5686290Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5687522Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5688744Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5689973Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5691193Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5691903Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5692390Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5692948Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5693443Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5694477Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5695714Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5696948Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5698180Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5699404Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5700625Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5701862Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5703092Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5704313Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5705537Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5706768Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5708044Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5709322Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5710548Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5711778Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5713000Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5714465Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5715670Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5716900Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5718114Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5718839Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5719330Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5719797Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5720282Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5721273Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5722502Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5723809Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5725314Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5726552Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5727760Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5728994Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5730216Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5731442Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5732653Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5733872Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5735084Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5736310Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5737513Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5738819Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5740108Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5741339Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5742558Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5743785Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5745008Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5745738Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5746205Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5746689Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5747172Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5748168Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5749385Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5750622Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5751850Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5753168Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5754407Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5755694Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5756918Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5758143Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5759402Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5760644Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5761865Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5763095Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5764527Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5765774Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5767003Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5768305Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5769542Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5770819Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5772042Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5773273Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5774493Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5775724Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5776942Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5778172Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5779391Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5780601Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5781829Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5783124Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5784356Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5785139Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5785628Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5786095Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5786577Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5787823Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:795: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:39:58.5788662Z   return torch._VF.split_with_sizes(self, split_size, dim)
2023-01-11T22:39:58.5789840Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:795: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:39:58.5790648Z   return torch._VF.split_with_sizes(self, split_size, dim)
2023-01-11T22:39:58.5791078Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5791566Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5791926Z dist init r=1, world=2
2023-01-11T22:39:58.5792161Z dist init r=0, world=2
2023-01-11T22:39:58.5792399Z ok (33.664s)
2023-01-11T22:39:58.5792835Z   test_diff_hyperparams_sharding_strategy_str_shard_grad_op (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2023-01-11T22:39:58.5793455Z Tests FSDP parity with DDP when using multiple parameter groups with ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74405
2023-01-11T22:39:58.5794001Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74406
2023-01-11T22:39:58.5794620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5795074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5795638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5796110Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5796690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5797120Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5797693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5798159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5798612Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5799098Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5799818Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5800526Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5801108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5801564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5802038Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5802525Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5802991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5803463Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5803940Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5804644Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5805104Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5805586Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5806054Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5806509Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5806987Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5807458Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5807923Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5808382Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5808862Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5809326Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5809801Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5810259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5810727Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5811200Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5811656Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5812125Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5812594Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5813063Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5813524Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5813994Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5814466Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5814922Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5815386Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5815859Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5816328Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5816864Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5817347Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5817869Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5818324Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5818795Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5819262Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5819734Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5820182Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5820646Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5821119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5821569Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5822044Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5822515Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5822984Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5823439Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5823906Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5824374Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5825666Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:39:58.5826443Z   warnings.warn(
2023-01-11T22:39:58.5827607Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:782: UserWarning: The passed-in `module` is on CPU and will thus have FSDP's sharding initialization run on CPU, which may be slower than on GPU. We recommend passing in the `device_id` argument for FSDP to move `module` to GPU for the sharding initialization. `module` must also be on GPU device to work with the `sync_module_states=True` flag since that requires GPU communication.
2023-01-11T22:39:58.5828386Z   warnings.warn(
2023-01-11T22:39:58.5828765Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5829250Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5829717Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5830193Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5830664Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5831119Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5831600Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5832073Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5832542Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5833059Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5833541Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5834060Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5834513Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5834991Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5835462Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5835929Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5836385Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5836858Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5837329Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5837781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5838258Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5838731Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5839200Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5839655Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5840127Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5840597Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5841071Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5841528Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5842002Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5842469Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5842916Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5843383Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5843854Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5844541Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5845006Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5845478Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5845945Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5846405Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5846878Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5847344Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5847808Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5848259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5848730Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5849202Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5849728Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5850214Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5850740Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5851210Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5851549Z dist init r=0, world=2
2023-01-11T22:39:58.5851801Z dist init r=1, world=2
2023-01-11T22:39:58.5852044Z ok (34.064s)
2023-01-11T22:39:58.5852405Z   test_diff_trainability (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2023-01-11T22:39:58.5853003Z Tests FSDP parity with DDP when using multiple parameter groups and ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74488
2023-01-11T22:39:58.5853546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74489
2023-01-11T22:39:58.5854179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5854616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5855202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5855673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5856233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5856677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5857248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5857713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5858156Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5858660Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5859363Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5860061Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5860573Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5861052Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5862059Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5863319Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5864570Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5865854Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5867098Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5868390Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5869619Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5870851Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5872080Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5873321Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5873930Z dist init r=1, world=2
2023-01-11T22:39:58.5874164Z dist init r=0, world=2
2023-01-11T22:39:58.5874406Z ok (7.417s)
2023-01-11T22:39:58.5874790Z   test_multiple_optimizers (__main__.TestFSDPUseOrigParamsMultipleParamGroups)
2023-01-11T22:39:58.5875356Z Tests using two optimizers where only one sets gradients to ``None``. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74571
2023-01-11T22:39:58.5875893Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74572
2023-01-11T22:39:58.5876507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5876944Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5877520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5877999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5878580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5879016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5879585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5880052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5880510Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5880996Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5881660Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5882410Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5882926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5883449Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5884700Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:39:58.5886275Z [W reducer.cpp:1310] Warning: find_unused_parameters=True was specified in DDP constructor, but did not find any unused parameters in the forward pass. This flag results in an extra traversal of the autograd graph every iteration,  which can adversely affect performance. If your model indeed never has any unused parameters in the forward pass, consider turning this flag off. Note that this warning may be a false positive if your model has flow control causing later iterations to have unused parameters. (function operator())
2023-01-11T22:39:58.5887131Z dist init r=0, world=2
2023-01-11T22:39:58.5887382Z dist init r=1, world=2
2023-01-11T22:39:58.5887623Z ok (4.713s)
2023-01-11T22:39:58.5887915Z   test_no_sync (__main__.TestFSDPUseOrigParamsNoSync)
2023-01-11T22:39:58.5888436Z Tests a basic ``no_sync()`` setup by comparing ``use_orig_params=True`` ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74654
2023-01-11T22:39:58.5888978Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74655
2023-01-11T22:39:58.5889592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5890045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5890624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5891098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5891662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5892113Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5892684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5893151Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5893590Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5894099Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5894759Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5895440Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5895963Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5896438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5896796Z dist init r=0, world=2
2023-01-11T22:39:58.5897030Z dist init r=1, world=2
2023-01-11T22:39:58.5897268Z ok (3.810s)
2023-01-11T22:39:58.5897632Z   test_access_params_after_forward (__main__.TestFSDPUseOrigParamsParamAccess)
2023-01-11T22:39:58.5898285Z Tests that accessing the original parameters after the forward but ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74737
2023-01-11T22:39:58.5898838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74738
2023-01-11T22:39:58.5899512Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5899965Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5900520Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5900991Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5901569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5901994Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5902570Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5903037Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5903490Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5903977Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5904639Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5905328Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5905856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5906311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5906853Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5907346Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5907818Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5908298Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5908781Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5909259Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5909355Z dist init r=0, world=2
2023-01-11T22:39:58.5909463Z dist init r=1, world=2
2023-01-11T22:39:58.5909564Z ok (3.810s)
2023-01-11T22:39:58.5909828Z   test_multiple_forward_offload_params_False (__main__.TestFSDPUseOrigParamsUnshardReshard)
2023-01-11T22:39:58.5910139Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74820
2023-01-11T22:39:58.5910361Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74821
2023-01-11T22:39:58.5910748Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5910925Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5911289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5911482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5911848Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5912022Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5912459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5912657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5912903Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5913200Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5913606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5913990Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5914224Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5914447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5915209Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5915952Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5916701Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5917437Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5918193Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5918928Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5919679Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5920411Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5921150Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5921932Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5922726Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5923453Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5924405Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5925171Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5925284Z dist init r=1, world=2
2023-01-11T22:39:58.5925393Z dist init r=0, world=2
2023-01-11T22:39:58.5925493Z ok (5.012s)
2023-01-11T22:39:58.5925738Z   test_multiple_forward_offload_params_True (__main__.TestFSDPUseOrigParamsUnshardReshard)
2023-01-11T22:39:58.5926049Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74903
2023-01-11T22:39:58.5926277Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74904
2023-01-11T22:39:58.5926655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5926833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5927216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5927410Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5927774Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5927948Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5928306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5928501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5928751Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5928999Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5929399Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5929797Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5930028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5930256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5931087Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5931881Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5932625Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5933361Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5934108Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5934844Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5935590Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5936321Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5937063Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5937799Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5938510Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5939238Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5940028Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5940767Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5940928Z dist init r=0, world=2
2023-01-11T22:39:58.5941035Z dist init r=1, world=2
2023-01-11T22:39:58.5941135Z ok (4.912s)
2023-01-11T22:39:58.5941415Z   test_summon_between_two_forwards_offload_params_False (__main__.TestFSDPUseOrigParamsUnshardReshard)
2023-01-11T22:39:58.5941725Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 74986
2023-01-11T22:39:58.5941949Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 74987
2023-01-11T22:39:58.5942330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5942507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5942871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5943063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5943428Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5943601Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5943972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5944164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5944419Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5944663Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5945051Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5945451Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5945682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5945911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5946662Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5947399Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5948140Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5948925Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5949680Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5950462Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5951212Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5951944Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5952669Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5953395Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5954130Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5954859Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5955599Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5956327Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5956443Z dist init r=0, world=2
2023-01-11T22:39:58.5956551Z dist init r=1, world=2
2023-01-11T22:39:58.5956650Z ok (5.312s)
2023-01-11T22:39:58.5956910Z   test_summon_between_two_forwards_offload_params_True (__main__.TestFSDPUseOrigParamsUnshardReshard)
2023-01-11T22:39:58.5957220Z Tests that ``use_orig_params=True`` has parity with ``False`` when ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75069
2023-01-11T22:39:58.5957443Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75070
2023-01-11T22:39:58.5957869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5958050Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5958523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5958717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5959082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5959301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5959663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5959854Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5960111Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5960360Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5960768Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5961166Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5961400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5961631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5962387Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5963128Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5963879Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5964828Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5965588Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5966325Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5967161Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5967896Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5968686Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5969409Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5970147Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5970877Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5971616Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5972345Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:39:58.5972462Z dist init r=0, world=2
2023-01-11T22:39:58.5972569Z dist init r=1, world=2
2023-01-11T22:39:58.5972668Z ok (5.414s)
2023-01-11T22:39:58.5972880Z   test_grad_writeback (__main__.TestFSDPUseOrigParamsWriteback)
2023-01-11T22:39:58.5973330Z Tests that changes to the original parameters' gradients are written ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75152
2023-01-11T22:39:58.5973551Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75153
2023-01-11T22:39:58.5973929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5974088Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5974470Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5974669Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5975036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5975209Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5975584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5975777Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5976026Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5976328Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5976724Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5977181Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5977415Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5977645Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5977881Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5978117Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5978350Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5978580Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5978798Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5979031Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5979260Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5979487Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5979710Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5979939Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5980166Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5980394Z INFO:torch.nn.parallel.distributed:Reducer buckets have been rebuilt in this iteration.
2023-01-11T22:39:58.5980507Z dist init r=1, world=2
2023-01-11T22:39:58.5980601Z dist init r=0, world=2
2023-01-11T22:39:58.5980700Z ok (3.910s)
2023-01-11T22:39:58.5980914Z   test_param_writeback (__main__.TestFSDPUseOrigParamsWriteback)
2023-01-11T22:39:58.5981231Z Tests that changes to the original parameters are written back. ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75235
2023-01-11T22:39:58.5981450Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75236
2023-01-11T22:39:58.5981831Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5982007Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5982387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5982561Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5982933Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5983106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5983490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5983679Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5983927Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5984172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5984574Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5984957Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5985242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5985483Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5985639Z dist init r=0, world=2
2023-01-11T22:39:58.5985749Z dist init r=1, world=2
2023-01-11T22:39:58.5985849Z ok (3.309s)
2023-01-11T22:39:58.5986199Z   test_writeback_shape_mismatch (__main__.TestFSDPUseOrigParamsWriteback) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75314
2023-01-11T22:39:58.5986421Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75315
2023-01-11T22:39:58.5986780Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5986955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5987341Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5987534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5987903Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:39:58.5988079Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:39:58.5988454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:39:58.5988644Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:39:58.5988889Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:39:58.5989118Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:39:58.5989521Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5989922Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:39:58.5990159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:39:58.5990386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:39:58.5990498Z dist init r=1, world=2
2023-01-11T22:39:58.5990606Z dist init r=0, world=2
2023-01-11T22:39:58.5990706Z ok (3.310s)
2023-01-11T22:39:58.5990727Z 
2023-01-11T22:39:58.5990980Z ----------------------------------------------------------------------
2023-01-11T22:39:58.5991096Z Ran 19 tests in 174.074s
2023-01-11T22:39:58.5991115Z 
2023-01-11T22:39:58.5991208Z OK
2023-01-11T22:39:58.5991226Z 
2023-01-11T22:39:58.5991351Z Generating XML reports...
2023-01-11T22:39:58.5991854Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsFQNs-20230111223704.xml
2023-01-11T22:39:58.5992416Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsMultipleParamGroups-20230111223704.xml
2023-01-11T22:39:58.5992913Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsNoSync-20230111223704.xml
2023-01-11T22:39:58.5993425Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsParamAccess-20230111223704.xml
2023-01-11T22:39:58.5993950Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsUnshardReshard-20230111223704.xml
2023-01-11T22:39:58.5994455Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsWriteback-20230111223704.xml
2023-01-11T22:39:58.5994476Z 
2023-01-11T22:39:58.5994931Z ##[endgroup]
2023-01-11T22:39:58.5995506Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_use_orig_params (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_use_orig_params_4hs5pk_y)
2023-01-11T22:39:58.5995567Z 
2023-01-11T22:39:58.5995867Z Running distributed/fsdp/test_fsdp_mixed_precision ... [2023-01-11 22:39:58.501044]
2023-01-11T22:39:58.5996362Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/fsdp/test_fsdp_mixed_precision.py', '-v', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:39:58.501324]
2023-01-11T22:44:53.2155420Z 
2023-01-11T22:44:53.2157833Z Expand the folded group to see the log file of distributed/fsdp/test_fsdp_mixed_precision
2023-01-11T22:44:53.2158838Z ##[group]PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_l17a2ety)
2023-01-11T22:44:53.2159239Z 
2023-01-11T22:44:53.2159333Z Running tests...
2023-01-11T22:44:53.2164735Z ----------------------------------------------------------------------
2023-01-11T22:44:53.2165367Z Test results will be stored in test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision
2023-01-11T22:44:53.2166050Z   test_float16_on_one_submodule (__main__.TestFSDPDifferentSubmodulePrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75428
2023-01-11T22:44:53.2166641Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75429
2023-01-11T22:44:53.2167263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2167727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2168319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2168782Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2169376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2169829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2170415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2170883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2171344Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2171849Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2172579Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2173277Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2173864Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2174746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2175425Z dist init r=1, world=2
2023-01-11T22:44:53.2175853Z dist init r=0, world=2
2023-01-11T22:44:53.2176319Z ok (4.727s)
2023-01-11T22:44:53.2177294Z   test_float16_on_one_submodule_skip_inputs (__main__.TestFSDPDifferentSubmodulePrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75512
2023-01-11T22:44:53.2178448Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75513
2023-01-11T22:44:53.2179109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2179569Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2180413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2181047Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2181957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2182560Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2183126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2183606Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2184068Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2184574Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2185320Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2186392Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2187506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2188445Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2188801Z dist init r=1, world=2
2023-01-11T22:44:53.2189056Z dist init r=0, world=2
2023-01-11T22:44:53.2189299Z ok (4.712s)
2023-01-11T22:44:53.2189793Z   test_float16_on_one_submodule_skip_inputs_error (__main__.TestFSDPDifferentSubmodulePrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75596
2023-01-11T22:44:53.2190838Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75597
2023-01-11T22:44:53.2191982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2192999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2194191Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2195044Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2196175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2197106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2198235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2199090Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2199937Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2200862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2202084Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2203311Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2204718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2205626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2206284Z dist init r=1, world=2
2023-01-11T22:44:53.2206732Z dist init r=0, world=2
2023-01-11T22:44:53.2207129Z ok (4.612s)
2023-01-11T22:44:53.2208004Z   test_submodules_with_different_precisions (__main__.TestFSDPDifferentSubmodulePrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75676
2023-01-11T22:44:53.2209227Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75677
2023-01-11T22:44:53.2210431Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2211378Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2212475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2213378Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2214476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2215265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2216337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2217214Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2218041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2218921Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2220052Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2221286Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2222172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2223003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2223573Z dist init r=1, world=2
2023-01-11T22:44:53.2223978Z dist init r=0, world=2
2023-01-11T22:44:53.2224478Z ok (4.712s)
2023-01-11T22:44:53.2225366Z   test_submodules_with_different_precisions_error (__main__.TestFSDPDifferentSubmodulePrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75760
2023-01-11T22:44:53.2226003Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75761
2023-01-11T22:44:53.2226638Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2227096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2227654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2228128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2228709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2229143Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2229725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2230197Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2230661Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2231160Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2233551Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2234289Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2234828Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2235294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2235755Z dist init r=0, world=2
2023-01-11T22:44:53.2236024Z dist init r=1, world=2
2023-01-11T22:44:53.2236247Z ok (4.712s)
2023-01-11T22:44:53.2236748Z   test_submodules_with_external_inputs (__main__.TestFSDPDifferentSubmodulePrecision) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75840
2023-01-11T22:44:53.2237393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75841
2023-01-11T22:44:53.2238022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2238465Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2239046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2239523Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2240112Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2240544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2241122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2241598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2242039Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2242537Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2243202Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2243900Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2244976Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2245470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2245833Z dist init r=0, world=2
2023-01-11T22:44:53.2246067Z dist init r=1, world=2
2023-01-11T22:44:53.2246303Z ok (4.612s)
2023-01-11T22:44:53.2246813Z   test_mixed_precision_with_ignored_module (__main__.TestFSDPMixedPrecisionIgnoredModules) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75924
2023-01-11T22:44:53.2247588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2248031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2248612Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2249084Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2249531Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2250198Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:44:53.2250747Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2251546Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2023-01-11T22:44:53.2252019Z   warnings.warn(
2023-01-11T22:44:53.2252274Z dist init r=0, world=1
2023-01-11T22:44:53.2252519Z ok (3.909s)
2023-01-11T22:44:53.2252976Z   test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 75964
2023-01-11T22:44:53.2253648Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 75965
2023-01-11T22:44:53.2254301Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2254833Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2255439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2255897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2256485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2256939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2257499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2257969Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2258434Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2258936Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2259590Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2260287Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2260820Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2261301Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2261642Z dist init r=0, world=2
2023-01-11T22:44:53.2261893Z dist init r=1, world=2
2023-01-11T22:44:53.2262131Z ok (4.912s)
2023-01-11T22:44:53.2262485Z   test_input_grads_with_param_mixed_precision (__main__.TestFSDPMixedPrecisionSharded)
2023-01-11T22:44:53.2263076Z Tests that input tensors that require gradients do get their gradients ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76048
2023-01-11T22:44:53.2264290Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76049
2023-01-11T22:44:53.2264924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2265367Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2265949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2266426Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2266994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2267446Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2268033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2268504Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2268948Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2269449Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2270110Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2270803Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2271330Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2271885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2272320Z dist init r=0, world=2
2023-01-11T22:44:53.2272587Z dist init r=1, world=2
2023-01-11T22:44:53.2272806Z ok (4.812s)
2023-01-11T22:44:53.2273421Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76132
2023-01-11T22:44:53.2274085Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76133
2023-01-11T22:44:53.2274704Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2275140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2275718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2276193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2276760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2277213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2277791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2278261Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2278699Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2299673Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2300407Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2301126Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2301644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2302132Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2303156Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2304435Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2305684Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2307205Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2308489Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2309890Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2311202Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2312440Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2313701Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2314934Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2316163Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2317412Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2318887Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2320282Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2321517Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2322749Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2323984Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2325864Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2326541Z dist init r=0, world=2
2023-01-11T22:44:53.2326799Z dist init r=1, world=2
2023-01-11T22:44:53.2327022Z ok (5.112s)
2023-01-11T22:44:53.2327567Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76216
2023-01-11T22:44:53.2328201Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76217
2023-01-11T22:44:53.2328836Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2329278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2329870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2330349Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2330925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2331376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2331951Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2332419Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2332862Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2333372Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2334050Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2334753Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2335270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2335752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2336760Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2337998Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2339247Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2340479Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2341787Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2343037Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2344324Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2345558Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2346800Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2348037Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2349278Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2350500Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2351741Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2352978Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2354218Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2355451Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2356729Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2357973Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2359275Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2360508Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2361729Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2362964Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2363569Z dist init r=1, world=2
2023-01-11T22:44:53.2363825Z dist init r=0, world=2
2023-01-11T22:44:53.2364048Z ok (5.012s)
2023-01-11T22:44:53.2365147Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76300
2023-01-11T22:44:53.2365825Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76301
2023-01-11T22:44:53.2366458Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2366901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2367485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2367965Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2368549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2368981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2369558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2370025Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2370472Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2370987Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2371650Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2372395Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2372913Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2373476Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2374506Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2375824Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2377073Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2378325Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2379555Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2380802Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2382034Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2383281Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2384516Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2385752Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2386987Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2388229Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2389488Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2390768Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2391999Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2393243Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2394473Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2395708Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2396327Z dist init r=1, world=2
2023-01-11T22:44:53.2396564Z dist init r=0, world=2
2023-01-11T22:44:53.2396804Z ok (5.112s)
2023-01-11T22:44:53.2397346Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76384
2023-01-11T22:44:53.2397983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76385
2023-01-11T22:44:53.2398589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2399049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2399629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2400086Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2400675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2401128Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2401709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2402159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2402621Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2403128Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2403793Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2405065Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2405697Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2406193Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2407261Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2408515Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2409776Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2411019Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2412253Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2413485Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2414737Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2415976Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2417218Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2418438Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2419681Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2420961Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2422240Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2423468Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2424707Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2425934Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2427175Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2428407Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2429627Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2430853Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2432093Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2433325Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2433932Z dist init r=0, world=2
2023-01-11T22:44:53.2434184Z dist init r=1, world=2
2023-01-11T22:44:53.2434405Z ok (5.113s)
2023-01-11T22:44:53.2434975Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76468
2023-01-11T22:44:53.2435682Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76469
2023-01-11T22:44:53.2436302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2436807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2437389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2437865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2438430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2438881Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2439454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2439904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2440367Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2440875Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2441544Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2442222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2442752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2443231Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2444730Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:44:53.2445526Z   return iter(self.unbind(0))
2023-01-11T22:44:53.2446649Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:44:53.2447437Z   return iter(self.unbind(0))
2023-01-11T22:44:53.2447708Z dist init r=1, world=2
2023-01-11T22:44:53.2447959Z dist init r=0, world=2
2023-01-11T22:44:53.2448179Z ok (5.112s)
2023-01-11T22:44:53.2448721Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76552
2023-01-11T22:44:53.2449352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76553
2023-01-11T22:44:53.2449956Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2450414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2450999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2451477Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2452046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2452502Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2453137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2453602Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2454100Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2454608Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2455277Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2455955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2456485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2456962Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2457976Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2459229Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2460470Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2461711Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2462955Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2464193Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2465435Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2466682Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2467922Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2469199Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2470482Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2471712Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2473009Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2474239Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2475479Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2476713Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2477957Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2479184Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2480422Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2481639Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2482879Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2484148Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2484976Z dist init r=1, world=2
2023-01-11T22:44:53.2485236Z dist init r=0, world=2
2023-01-11T22:44:53.2485459Z ok (5.112s)
2023-01-11T22:44:53.2486030Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76636
2023-01-11T22:44:53.2486692Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76637
2023-01-11T22:44:53.2487318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2487763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2488348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2488827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2489390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2489836Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2490413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2490880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2491324Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2491833Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2492501Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2493205Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2493722Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2494201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2495448Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:44:53.2496248Z   return iter(self.unbind(0))
2023-01-11T22:44:53.2497370Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:44:53.2498163Z   return iter(self.unbind(0))
2023-01-11T22:44:53.2498431Z dist init r=0, world=2
2023-01-11T22:44:53.2498685Z dist init r=1, world=2
2023-01-11T22:44:53.2498908Z ok (5.112s)
2023-01-11T22:44:53.2499444Z   test_mixed_precision_e2e_full_shard_mp_diff_buffer_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76720
2023-01-11T22:44:53.2500147Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76721
2023-01-11T22:44:53.2500785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2501291Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2501877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2502351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2502916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2503366Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2503948Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2504423Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2504866Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2505369Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2506041Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2506746Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2507263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2507741Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2508753Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2509998Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2511248Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2512482Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2513743Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2514985Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2516273Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2517515Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2518800Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2520045Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2521290Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2522505Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2523755Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2525176Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2526419Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2527653Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2528893Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2530137Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2531443Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2532699Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2533987Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2535224Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2535836Z dist init r=1, world=2
2023-01-11T22:44:53.2536091Z dist init r=0, world=2
2023-01-11T22:44:53.2536316Z ok (5.112s)
2023-01-11T22:44:53.2536868Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76804
2023-01-11T22:44:53.2537509Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76805
2023-01-11T22:44:53.2538137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2538579Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2539171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2539655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2540243Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2540676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2541257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2541730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2542172Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2542682Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2543357Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2544064Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2544577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2545062Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2546063Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2547328Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2548613Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2549908Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2551122Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2552373Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2553619Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2554854Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2556088Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2557325Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2558557Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2559789Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2561030Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2562252Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2563540Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2565008Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2566236Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2567480Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2568095Z dist init r=0, world=2
2023-01-11T22:44:53.2568333Z dist init r=1, world=2
2023-01-11T22:44:53.2568576Z ok (5.012s)
2023-01-11T22:44:53.2569100Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76888
2023-01-11T22:44:53.2569713Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76889
2023-01-11T22:44:53.2570312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2570767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2571354Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2571812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2572445Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2572899Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2573478Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2573932Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2574388Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2574895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2575564Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2576247Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2576783Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2577263Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2578267Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2579589Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2580921Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2582234Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2583480Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2584713Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2585965Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2587199Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2588444Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2589663Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2590905Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2592138Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2593379Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2594661Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2595913Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2597194Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2598435Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2599664Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2600886Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2602126Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2603366Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2604770Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2605386Z dist init r=1, world=2
2023-01-11T22:44:53.2605641Z dist init r=0, world=2
2023-01-11T22:44:53.2605862Z ok (5.112s)
2023-01-11T22:44:53.2606418Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 76972
2023-01-11T22:44:53.2607068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 76973
2023-01-11T22:44:53.2607677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2608137Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2608723Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2609201Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2609766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2610219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2610869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2611354Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2611855Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2612361Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2613032Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2613716Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2614245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2614729Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2615736Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2616999Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2618233Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2619487Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2620728Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2621966Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2623192Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2624438Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2625675Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2626962Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2628226Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2629470Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2630708Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2631950Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2633182Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2634421Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2635663Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2636901Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2637494Z dist init r=0, world=2
2023-01-11T22:44:53.2637748Z dist init r=1, world=2
2023-01-11T22:44:53.2637988Z ok (5.112s)
2023-01-11T22:44:53.2638510Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77056
2023-01-11T22:44:53.2639108Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77057
2023-01-11T22:44:53.2639726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2640186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2640750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2641234Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2641865Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2642324Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2642935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2643402Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2643860Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2644534Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2645196Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2645895Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2646429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2646890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2647895Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2649145Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2650402Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2651649Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2652898Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2654132Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2655379Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2656595Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2657934Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2659250Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2660477Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2661706Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2662942Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2664175Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2665415Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2666650Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2667876Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2669110Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2670351Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2671585Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2672919Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2674204Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2674821Z dist init r=1, world=2
2023-01-11T22:44:53.2675056Z dist init r=0, world=2
2023-01-11T22:44:53.2675295Z ok (5.112s)
2023-01-11T22:44:53.2675844Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77140
2023-01-11T22:44:53.2676484Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77141
2023-01-11T22:44:53.2677090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2677548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2678137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2678592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2679178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2679626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2680201Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2680649Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2681112Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2681623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2682272Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2682973Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2683507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2683987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2685395Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:44:53.2686183Z   return iter(self.unbind(0))
2023-01-11T22:44:53.2687314Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:44:53.2688108Z   return iter(self.unbind(0))
2023-01-11T22:44:53.2688375Z dist init r=1, world=2
2023-01-11T22:44:53.2688607Z dist init r=0, world=2
2023-01-11T22:44:53.2688843Z ok (5.112s)
2023-01-11T22:44:53.2689441Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77224
2023-01-11T22:44:53.2690062Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77225
2023-01-11T22:44:53.2690741Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2691196Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2691781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2692242Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2692828Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2693278Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2693856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2694312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2694775Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2695282Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2695926Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2696632Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2697165Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2697644Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2698634Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2699886Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2701137Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2702379Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2703627Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2704860Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2706151Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2707436Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2708680Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2709900Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2711145Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2712374Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2713605Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2714837Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2716079Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2717317Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2718557Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2719775Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2721061Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2722343Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2723579Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2724982Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2725600Z dist init r=1, world=2
2023-01-11T22:44:53.2725837Z dist init r=0, world=2
2023-01-11T22:44:53.2726078Z ok (5.114s)
2023-01-11T22:44:53.2726625Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77308
2023-01-11T22:44:53.2727264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77309
2023-01-11T22:44:53.2727872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2728331Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2728920Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2729398Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2729970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2730421Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2730999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2731453Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2731914Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2732420Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2733092Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2733773Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2734304Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2734780Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2736025Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:44:53.2736798Z   return iter(self.unbind(0))
2023-01-11T22:44:53.2738009Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:44:53.2738849Z   return iter(self.unbind(0))
2023-01-11T22:44:53.2739121Z dist init r=0, world=2
2023-01-11T22:44:53.2739355Z dist init r=1, world=2
2023-01-11T22:44:53.2739597Z ok (5.213s)
2023-01-11T22:44:53.2740112Z   test_mixed_precision_e2e_full_shard_mp_fp16_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77392
2023-01-11T22:44:53.2740719Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77393
2023-01-11T22:44:53.2741336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2741790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2742374Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2742851Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2743418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2743865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2744441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2744897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2745362Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2745867Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2746537Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2747215Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2747744Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2748222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2749228Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2750468Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2751726Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2753017Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2754280Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2755577Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2756822Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2758062Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2759309Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2760541Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2761764Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2763003Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2764405Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2765654Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2766898Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2768125Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2769429Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2770725Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2771944Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2773231Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2774460Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2775699Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2776307Z dist init r=1, world=2
2023-01-11T22:44:53.2776565Z dist init r=0, world=2
2023-01-11T22:44:53.2776784Z ok (5.213s)
2023-01-11T22:44:53.2777333Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77476
2023-01-11T22:44:53.2777984Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77477
2023-01-11T22:44:53.2778585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2779042Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2779630Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2780111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2780679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2781130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2781711Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2782184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2782623Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2783127Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2783796Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2784477Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2785063Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2785556Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2786615Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2787859Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2789090Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2790337Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2791592Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2792832Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2794075Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2795296Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2796542Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2797782Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2799021Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2800282Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2801592Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2802824Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2804058Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2805549Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2806792Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2808026Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2809267Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2810475Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2811086Z dist init r=0, world=2
2023-01-11T22:44:53.2811338Z dist init r=1, world=2
2023-01-11T22:44:53.2811583Z ok (5.112s)
2023-01-11T22:44:53.2812092Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77560
2023-01-11T22:44:53.2812712Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77561
2023-01-11T22:44:53.2813334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2813791Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2814355Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2814833Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2815421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2815928Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2816521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2817053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2817513Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2818000Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2818668Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2819368Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2819897Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2820363Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2821369Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2822635Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2823887Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2825133Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2826375Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2827603Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2828841Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2830084Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2831363Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2832618Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2833903Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2835142Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2836369Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2837614Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2838829Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2840066Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2841295Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2842527Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2843754Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2845185Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2845801Z dist init r=0, world=2
2023-01-11T22:44:53.2846037Z dist init r=1, world=2
2023-01-11T22:44:53.2846276Z ok (5.012s)
2023-01-11T22:44:53.2846904Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77644
2023-01-11T22:44:53.2847563Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77645
2023-01-11T22:44:53.2848229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2848686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2849271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2849731Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2850322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2850779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2851357Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2851808Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2852272Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2852784Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2853430Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2854129Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2854667Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2855142Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2856135Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2857397Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2858650Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2859884Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2861128Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2862365Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2863660Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2864952Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2866191Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2867403Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2868647Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2869879Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2871116Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2872391Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2873643Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2874878Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2876116Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2877329Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2878621Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2879954Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2880567Z dist init r=1, world=2
2023-01-11T22:44:53.2880821Z dist init r=0, world=2
2023-01-11T22:44:53.2881044Z ok (5.012s)
2023-01-11T22:44:53.2881567Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77728
2023-01-11T22:44:53.2882178Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77729
2023-01-11T22:44:53.2882800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2883243Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2883826Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2884522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2885106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2885555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2886127Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2886600Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2887041Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2887549Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2888220Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2888919Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2889427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2889909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2890921Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2892174Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2893408Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2894731Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2895994Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2897302Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2898542Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2899777Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2901007Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2902248Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2903490Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2904726Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2905948Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2907183Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2908412Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2909695Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2910937Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2912246Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2913479Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2914717Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2915313Z dist init r=1, world=2
2023-01-11T22:44:53.2915568Z dist init r=0, world=2
2023-01-11T22:44:53.2915808Z ok (5.012s)
2023-01-11T22:44:53.2916343Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77812
2023-01-11T22:44:53.2916981Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77813
2023-01-11T22:44:53.2917604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2918065Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2918635Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2919111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2919698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2920153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2920713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2921181Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2921644Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2922136Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2922808Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2923507Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2924035Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2924734Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2925746Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2927077Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2928389Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2929642Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2930891Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2932132Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2933357Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2934592Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2935831Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2937059Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2938302Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2939532Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2940771Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2942048Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2943336Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2944558Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2945805Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2947041Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2948272Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2949504Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2950122Z dist init r=0, world=2
2023-01-11T22:44:53.2950358Z dist init r=1, world=2
2023-01-11T22:44:53.2950598Z ok (5.112s)
2023-01-11T22:44:53.2951117Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77896
2023-01-11T22:44:53.2951726Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77897
2023-01-11T22:44:53.2952328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2952787Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2953376Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2953841Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2954426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2954873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2955450Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2955906Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2956363Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2956870Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2957591Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2958290Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2958871Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2959351Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2960360Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2961606Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2962855Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2964098Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2965565Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2966809Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2968049Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2969294Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2970539Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2971759Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2973119Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2974419Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2975651Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2976889Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2978120Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2979361Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2980586Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2981819Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2983028Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2984263Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2984879Z dist init r=0, world=2
2023-01-11T22:44:53.2985134Z dist init r=1, world=2
2023-01-11T22:44:53.2985357Z ok (5.112s)
2023-01-11T22:44:53.2985907Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 77980
2023-01-11T22:44:53.2986546Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 77981
2023-01-11T22:44:53.2987168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2987607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2988245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2988732Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2989324Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.2989809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.2990389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.2990858Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.2991300Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.2991814Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.2992480Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2993180Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.2993694Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.2994176Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.2995187Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2996443Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2997692Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.2998928Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3000159Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3001397Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3002648Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3003928Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3005424Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3006744Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3007992Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3009224Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3010468Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3011688Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3012931Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3014163Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3015400Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3016634Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3017874Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3019168Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3019793Z dist init r=1, world=2
2023-01-11T22:44:53.3020031Z dist init r=0, world=2
2023-01-11T22:44:53.3020272Z ok (5.112s)
2023-01-11T22:44:53.3020858Z   test_mixed_precision_e2e_full_shard_mp_no_mp_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78064
2023-01-11T22:44:53.3021447Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78065
2023-01-11T22:44:53.3022072Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3022531Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3023095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3023572Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3024158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3024611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3025173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3025647Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3026107Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3026620Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3027272Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3027971Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3028507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3028966Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3029975Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3031227Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3032474Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3033727Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3034967Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3036265Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3037549Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3038772Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3040006Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3041249Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3042477Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3043722Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3045191Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3046430Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3047657Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3048901Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3050114Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3051426Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3052722Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3053953Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3054561Z dist init r=0, world=2
2023-01-11T22:44:53.3054814Z dist init r=1, world=2
2023-01-11T22:44:53.3055041Z ok (5.012s)
2023-01-11T22:44:53.3055613Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78148
2023-01-11T22:44:53.3056276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78149
2023-01-11T22:44:53.3056881Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3057337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3057922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3058395Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3058959Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3059415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3059994Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3060450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3060907Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3061411Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3062078Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3062757Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3063281Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3063764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3064770Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3066022Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3067305Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3068554Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3069858Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3071100Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3072352Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3073639Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3074879Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3076106Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3077332Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3078572Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3079812Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3081041Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3082321Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3083567Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3085088Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3086335Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3086927Z dist init r=0, world=2
2023-01-11T22:44:53.3087181Z dist init r=1, world=2
2023-01-11T22:44:53.3087423Z ok (5.012s)
2023-01-11T22:44:53.3087963Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78232
2023-01-11T22:44:53.3088572Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78233
2023-01-11T22:44:53.3089189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3089371Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3089763Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3089961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3090335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3090496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3090880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3091068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3091319Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3091568Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3091974Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3092381Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3092616Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3092853Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3093596Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3094353Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3095167Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3095979Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3096713Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3097467Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3098199Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3098934Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3099664Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3100416Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3101148Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3101891Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3102627Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3103368Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3104148Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3104936Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3105667Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3106412Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3107147Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3107887Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3108617Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3109360Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3109475Z dist init r=0, world=2
2023-01-11T22:44:53.3109584Z dist init r=1, world=2
2023-01-11T22:44:53.3109684Z ok (5.112s)
2023-01-11T22:44:53.3110128Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78316
2023-01-11T22:44:53.3110352Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78317
2023-01-11T22:44:53.3110735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3110900Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3111287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3111481Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3111851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3112031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3112410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3112653Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3112913Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3113215Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3113605Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3114007Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3114240Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3114471Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3115227Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3115972Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3116718Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3117459Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3118206Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3118944Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3119695Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3120432Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3121172Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3121958Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3122748Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3123478Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3124438Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3125199Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3125938Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3126668Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3127413Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3128146Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3128263Z dist init r=0, world=2
2023-01-11T22:44:53.3128371Z dist init r=1, world=2
2023-01-11T22:44:53.3128453Z ok (5.112s)
2023-01-11T22:44:53.3128869Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78400
2023-01-11T22:44:53.3129095Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78401
2023-01-11T22:44:53.3129475Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3129653Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3130039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3130236Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3130604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3130854Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3131232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3131487Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3131741Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3131988Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3132396Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3132796Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3133031Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3133264Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3134018Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3134779Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3135520Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3136261Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3136999Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3137732Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3138480Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3139215Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3140010Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3140758Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3141549Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3142287Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3143030Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3143752Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3144497Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3145224Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3145964Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3146697Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3147439Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3148174Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3149002Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3149747Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3149911Z dist init r=1, world=2
2023-01-11T22:44:53.3150019Z dist init r=0, world=2
2023-01-11T22:44:53.3150120Z ok (5.112s)
2023-01-11T22:44:53.3150562Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78484
2023-01-11T22:44:53.3150783Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78485
2023-01-11T22:44:53.3151165Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3151350Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3151717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3151915Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3152280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3152456Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3152835Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3153026Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3153277Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3153527Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3153937Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3154324Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3154558Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3154789Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3155789Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:44:53.3155922Z   return iter(self.unbind(0))
2023-01-11T22:44:53.3156898Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:44:53.3157029Z   return iter(self.unbind(0))
2023-01-11T22:44:53.3157143Z dist init r=1, world=2
2023-01-11T22:44:53.3157253Z dist init r=0, world=2
2023-01-11T22:44:53.3157353Z ok (5.212s)
2023-01-11T22:44:53.3157743Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78568
2023-01-11T22:44:53.3158020Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78569
2023-01-11T22:44:53.3158406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3158630Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3159017Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3159213Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3159581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3159757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3160136Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3160312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3160566Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3160816Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3161222Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3161624Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3161860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3162090Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3162847Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3163588Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3164559Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3165312Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3166066Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3166806Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3167623Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3168424Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3169171Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3169908Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3170648Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3171385Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3172130Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3172905Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3173651Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3174388Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3175129Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3175864Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3176650Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3177391Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3178209Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3178946Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3179043Z dist init r=0, world=2
2023-01-11T22:44:53.3179157Z dist init r=1, world=2
2023-01-11T22:44:53.3179258Z ok (5.014s)
2023-01-11T22:44:53.3179699Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78652
2023-01-11T22:44:53.3179970Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78653
2023-01-11T22:44:53.3180352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3180534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3180922Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3181097Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3181468Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3181648Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3182035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3182229Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3182478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3182727Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3183134Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3183541Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3183757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3183992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3184988Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:44:53.3185119Z   return iter(self.unbind(0))
2023-01-11T22:44:53.3186148Z /opt/conda/lib/python3.10/site-packages/torch/_tensor.py:932: UserWarning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (Triggered internally at /var/lib/jenkins/workspace/torch/csrc/autograd/python_variable.cpp:319.)
2023-01-11T22:44:53.3186327Z   return iter(self.unbind(0))
2023-01-11T22:44:53.3186441Z dist init r=0, world=2
2023-01-11T22:44:53.3186553Z dist init r=1, world=2
2023-01-11T22:44:53.3186654Z ok (5.112s)
2023-01-11T22:44:53.3187060Z   test_mixed_precision_e2e_full_shard_mp_only_param_and_buf_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78736
2023-01-11T22:44:53.3187281Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78737
2023-01-11T22:44:53.3187649Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3187829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3188211Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3188408Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3188776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3188954Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3189338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3189528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3189760Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3190011Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3190417Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3190822Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3191056Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3191288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3192048Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3192794Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3193547Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3194284Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3195087Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3195876Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3196606Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3197337Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3198090Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3198824Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3199567Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3200305Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3201041Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3201774Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3202519Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3203252Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3204041Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3205062Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3205807Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3206546Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3207286Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3208012Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3208127Z dist init r=1, world=2
2023-01-11T22:44:53.3208220Z dist init r=0, world=2
2023-01-11T22:44:53.3208322Z ok (5.112s)
2023-01-11T22:44:53.3208756Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78820
2023-01-11T22:44:53.3208983Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78821
2023-01-11T22:44:53.3209362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3209540Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3209930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3210123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3210477Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3210655Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3211033Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3211228Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3211478Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3211724Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3212131Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3212532Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3212843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3213067Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3213826Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3214648Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3215392Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3216138Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3216878Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3217625Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3218363Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3219112Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3219846Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3220589Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3221327Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3222114Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3222856Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3223649Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3224385Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3225126Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3225858Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3226602Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3227335Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3228075Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3228188Z dist init r=1, world=2
2023-01-11T22:44:53.3228296Z dist init r=0, world=2
2023-01-11T22:44:53.3228396Z ok (5.012s)
2023-01-11T22:44:53.3228803Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78904
2023-01-11T22:44:53.3229031Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78905
2023-01-11T22:44:53.3229413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3229572Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3229960Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3230155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3230523Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3230700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3231132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3231332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3231631Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3231859Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3232271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3232674Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3232909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3233144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3233899Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3234638Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3235383Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3236120Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3236868Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3237599Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3238345Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3239082Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3239823Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3240600Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3241405Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3242138Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3242879Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3243615Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3244579Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3245330Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3246072Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3246805Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3247545Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3248279Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3248397Z dist init r=1, world=2
2023-01-11T22:44:53.3248505Z dist init r=0, world=2
2023-01-11T22:44:53.3248588Z ok (5.012s)
2023-01-11T22:44:53.3249098Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 78988
2023-01-11T22:44:53.3249335Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 78989
2023-01-11T22:44:53.3249718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3249955Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3250348Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3250543Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3250914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3251089Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3251449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3251642Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3251892Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3252143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3252554Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3252955Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3253190Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3253423Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3254177Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3254932Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3255672Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3256425Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3257147Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3257896Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3258681Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3259476Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3260217Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3260959Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3261693Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3262438Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3263177Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3263921Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3264652Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3265395Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3266132Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3266874Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3267650Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3268439Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3268556Z dist init r=0, world=2
2023-01-11T22:44:53.3268665Z dist init r=1, world=2
2023-01-11T22:44:53.3268766Z ok (5.012s)
2023-01-11T22:44:53.3269169Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_false_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79072
2023-01-11T22:44:53.3269393Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79073
2023-01-11T22:44:53.3269775Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3269956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3270330Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3270525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3270894Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3271071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3271449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3271641Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3271895Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3272143Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3272579Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3272980Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3273215Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3273447Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3274199Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3274934Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3275686Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3276471Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3277233Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3278019Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3278767Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3279512Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3280252Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3280981Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3281720Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3282454Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3283195Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3283924Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3284877Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3285693Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3286445Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3287277Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3288021Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3288754Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3288871Z dist init r=1, world=2
2023-01-11T22:44:53.3288980Z dist init r=0, world=2
2023-01-11T22:44:53.3289080Z ok (5.012s)
2023-01-11T22:44:53.3289496Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79156
2023-01-11T22:44:53.3289717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79157
2023-01-11T22:44:53.3290102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3290283Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3290666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3290865Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3291235Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3291410Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3291787Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3291958Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3292206Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3292454Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3292864Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3293271Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3293507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3293738Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3294495Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3295297Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3296084Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3296822Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3297552Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3298303Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3299037Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3299790Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3300524Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3301269Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3302005Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3302751Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3303485Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3304269Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3305066Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3305808Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3306544Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3307284Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3307997Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3308742Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3308862Z dist init r=0, world=2
2023-01-11T22:44:53.3308972Z dist init r=1, world=2
2023-01-11T22:44:53.3309072Z ok (5.112s)
2023-01-11T22:44:53.3309474Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp32_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79240
2023-01-11T22:44:53.3309698Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79241
2023-01-11T22:44:53.3310076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3310254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3310645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3310821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3311199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3311375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3311753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3311946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3312194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3312441Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3312896Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3313311Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3313577Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3313809Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3314561Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3315301Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3316051Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3316791Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3317543Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3318276Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3319029Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3319767Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3320505Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3321242Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3322027Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3322772Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3323558Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3324506Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3325268Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3326001Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3326743Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3327467Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3328205Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3328937Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3329052Z dist init r=1, world=2
2023-01-11T22:44:53.3329162Z dist init r=0, world=2
2023-01-11T22:44:53.3329264Z ok (5.112s)
2023-01-11T22:44:53.3329697Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_enable_sharded_grad_scaler (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79324
2023-01-11T22:44:53.3329903Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79325
2023-01-11T22:44:53.3330284Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3330461Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3330847Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3331115Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3331502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3331743Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3332123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3332313Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3332544Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3332790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3333191Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3333602Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3333838Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3334072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3334826Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3335575Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3336308Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3337059Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3337791Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3338543Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3339282Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3340029Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3340814Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3341604Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3342337Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3343075Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3343803Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3344540Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3345277Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3346020Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3346750Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3347494Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3348227Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3348963Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3349124Z dist init r=1, world=2
2023-01-11T22:44:53.3349221Z dist init r=0, world=2
2023-01-11T22:44:53.3349327Z ok (5.112s)
2023-01-11T22:44:53.3349732Z   test_mixed_precision_e2e_full_shard_mp_only_reduce_offload_true_fp64_none (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79408
2023-01-11T22:44:53.3350000Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79409
2023-01-11T22:44:53.3350385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3350566Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3350950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3351144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3351499Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3351679Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3352063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3352254Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3352505Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3352754Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3353160Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3353563Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3353798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3354013Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3354774Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3355516Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3356269Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3357010Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3357758Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3358548Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3359348Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3360082Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3360825Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3361553Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3362290Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3363024Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3363763Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3364714Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3365473Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3366206Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3366947Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3367748Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3368562Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3369294Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3369410Z dist init r=1, world=2
2023-01-11T22:44:53.3369519Z dist init r=0, world=2
2023-01-11T22:44:53.3369619Z ok (5.112s)
2023-01-11T22:44:53.3369992Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79492
2023-01-11T22:44:53.3370217Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79493
2023-01-11T22:44:53.3370577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3370758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3371145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3371337Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3371705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3371882Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3372262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3372452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3372749Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3372981Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3373392Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3373795Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3374032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3374266Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3374380Z dist init r=1, world=2
2023-01-11T22:44:53.3374487Z dist init r=0, world=2
2023-01-11T22:44:53.3374586Z ok (4.712s)
2023-01-11T22:44:53.3374790Z   test_mixed_precision_resnet (__main__.TestFSDPMixedPrecisionSharded)
2023-01-11T22:44:53.3375019Z End to end test to ensure mixed precision + auto_wrap works ... skip: no torchvision (0.001s)
2023-01-11T22:44:53.3375380Z   test_mp_batchnorm_convert_sync_bn_False (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79576
2023-01-11T22:44:53.3375600Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79577
2023-01-11T22:44:53.3375980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3376158Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3376601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3376802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3377162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3377390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3377769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3377960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3378212Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3378457Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3378865Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3379271Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3379507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3379720Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3379833Z dist init r=1, world=2
2023-01-11T22:44:53.3379940Z dist init r=0, world=2
2023-01-11T22:44:53.3380040Z ok (4.713s)
2023-01-11T22:44:53.3380396Z   test_mp_batchnorm_convert_sync_bn_True (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79660
2023-01-11T22:44:53.3380618Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79661
2023-01-11T22:44:53.3380997Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3381177Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3381547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3381742Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3382113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3382288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3382667Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3382857Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3383103Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3383350Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3383751Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3384140Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3384374Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3384602Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3384714Z dist init r=0, world=2
2023-01-11T22:44:53.3384823Z dist init r=1, world=2
2023-01-11T22:44:53.3384923Z ok (4.713s)
2023-01-11T22:44:53.3385265Z   test_mp_embedding_default (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79744
2023-01-11T22:44:53.3385544Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79745
2023-01-11T22:44:53.3385915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3386145Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3386530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3386723Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3387089Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3387266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3387644Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3387832Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3388065Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3388314Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3388723Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3389121Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3389353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3389586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3389701Z dist init r=1, world=2
2023-01-11T22:44:53.3389809Z dist init r=0, world=2
2023-01-11T22:44:53.3389907Z ok (4.812s)
2023-01-11T22:44:53.3390250Z   test_mp_embedding_only_params_and_bufs (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79828
2023-01-11T22:44:53.3390474Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79829
2023-01-11T22:44:53.3390856Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3391034Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3391417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3391610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3391977Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3392153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3392515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3392706Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3392952Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3393202Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3393606Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3394007Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3394241Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3394474Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3394587Z dist init r=1, world=2
2023-01-11T22:44:53.3394729Z dist init r=0, world=2
2023-01-11T22:44:53.3394839Z ok (4.812s)
2023-01-11T22:44:53.3395206Z   test_mp_embedding_params_and_reduce_diff (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79912
2023-01-11T22:44:53.3395471Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79913
2023-01-11T22:44:53.3395853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3396031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3396415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3396608Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3396957Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3397136Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3397510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3397702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3397950Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3398194Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3398594Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3398992Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3399223Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3399438Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3399567Z dist init r=0, world=2
2023-01-11T22:44:53.3399677Z dist init r=1, world=2
2023-01-11T22:44:53.3399775Z ok (4.812s)
2023-01-11T22:44:53.3400117Z   test_mp_embedding_reduce (__main__.TestFSDPMixedPrecisionSharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 79996
2023-01-11T22:44:53.3400320Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 79997
2023-01-11T22:44:53.3400700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3400875Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3401260Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3401456Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3401821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3401997Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3402373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3402563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3402790Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 1
2023-01-11T22:44:53.3403033Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3403434Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3403884Z INFO:torch.distributed.distributed_c10d:Rank 1: Completed store-based barrier for key:store_based_barrier_key:1 with 2 nodes.
2023-01-11T22:44:53.3404125Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T22:44:53.3404630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3404746Z dist init r=1, world=2
2023-01-11T22:44:53.3404856Z dist init r=0, world=2
2023-01-11T22:44:53.3404938Z ok (4.913s)
2023-01-11T22:44:53.3405294Z   test_grads_reduced_precision (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80080
2023-01-11T22:44:53.3405680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3405856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3406240Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3406437Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3406680Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3407086Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:44:53.3407317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3407842Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2023-01-11T22:44:53.3407954Z   warnings.warn(
2023-01-11T22:44:53.3408713Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3409452Z [W python_variable.cpp:319] Warning: Deallocating Tensor that still has live PyObject references.  This probably happened because you took out a weak reference to Tensor and didn't call _fix_weakref() after dereferencing it.  Subsequent accesses to this tensor via the PyObject will now fail. (function decref)
2023-01-11T22:44:53.3409570Z dist init r=0, world=1
2023-01-11T22:44:53.3409669Z ok (4.109s)
2023-01-11T22:44:53.3410026Z   test_mixed_precision_e2e_full_shard (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80122
2023-01-11T22:44:53.3410402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3410581Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3410964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3411140Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3411389Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3411789Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:44:53.3412018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3412551Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.FULL_SHARD since the world size is 1.
2023-01-11T22:44:53.3412668Z   warnings.warn(
2023-01-11T22:44:53.3412778Z dist init r=0, world=1
2023-01-11T22:44:53.3412876Z ok (3.909s)
2023-01-11T22:44:53.3413311Z   test_mixed_precision_no_reshard_after_forward (__main__.TestFSDPMixedPrecisionUnsharded) ... INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80164
2023-01-11T22:44:53.3413705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T22:44:53.3413946Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T22:44:53.3414332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T22:44:53.3414525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T22:44:53.3414770Z INFO:torch.distributed.distributed_c10d:Added key: store_based_barrier_key:1 to store for rank: 0
2023-01-11T22:44:53.3415170Z INFO:torch.distributed.distributed_c10d:Rank 0: Completed store-based barrier for key:store_based_barrier_key:1 with 1 nodes.
2023-01-11T22:44:53.3415400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T22:44:53.3415949Z /opt/conda/lib/python3.10/site-packages/torch/distributed/fsdp/_init_utils.py:288: UserWarning: FSDP is switching to use `NO_SHARD` instead of ShardingStrategy.SHARD_GRAD_OP since the world size is 1.
2023-01-11T22:44:53.3416049Z   warnings.warn(
2023-01-11T22:44:53.3416160Z dist init r=0, world=1
2023-01-11T22:44:53.3416258Z ok (3.909s)
2023-01-11T22:44:53.3416284Z 
2023-01-11T22:44:53.3416553Z ----------------------------------------------------------------------
2023-01-11T22:44:53.3416669Z Ran 60 tests in 290.732s
2023-01-11T22:44:53.3416689Z 
2023-01-11T22:44:53.3416795Z OK (skipped=1)
2023-01-11T22:44:53.3416813Z 
2023-01-11T22:44:53.3416937Z Generating XML reports...
2023-01-11T22:44:53.3417484Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPDifferentSubmodulePrecision-20230111224002.xml
2023-01-11T22:44:53.3418024Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionIgnoredModules-20230111224002.xml
2023-01-11T22:44:53.3418519Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20230111224002.xml
2023-01-11T22:44:53.3419031Z Generated XML report: test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20230111224002.xml
2023-01-11T22:44:53.3419052Z 
2023-01-11T22:44:53.3419493Z ##[endgroup]
2023-01-11T22:44:53.3420005Z FINISHED PRINTING LOG FILE of distributed/fsdp/test_fsdp_mixed_precision (/var/lib/jenkins/workspace/test/test-reports/distributed-fsdp-test_fsdp_mixed_precision_l17a2ety)
2023-01-11T22:44:53.3420026Z 
2023-01-11T22:44:53.3420319Z Running distributed/rpc/cuda/test_tensorpipe_agent ... [2023-01-11 22:44:53.217767]
2023-01-11T22:44:53.3420854Z Executing ['/opt/conda/bin/python', '-bb', 'distributed/rpc/cuda/test_tensorpipe_agent.py', '-v', '--subprocess', '--import-slow-tests', '--import-disabled-tests'] ... [2023-01-11 22:44:53.218155]
2023-01-11T23:03:32.2725320Z 
2023-01-11T23:03:32.2726083Z Expand the folded group to see the log file of distributed/rpc/cuda/test_tensorpipe_agent
2023-01-11T23:03:32.2730357Z ##[group]PRINTING LOG FILE of distributed/rpc/cuda/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-cuda-test_tensorpipe_agent_mppcsvm1)
2023-01-11T23:03:32.2731050Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy2u0do5o
2023-01-11T23:03:32.2731611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy2u0do5o/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2732223Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaDdpComparisonTest testMethod=test_ddp_dist_autograd_local_vs_remote_gpu>]>
2023-01-11T23:03:32.2732810Z test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest)
2023-01-11T23:03:32.2733884Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_simple>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation>, <__main__.TensorPipeCudaDistAutogradTest testMethod=test_gpu_to_cpu_continuation_gpu_root>]>
2023-01-11T23:03:32.2734753Z test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest)
2023-01-11T23:03:32.2735196Z test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest)
2023-01-11T23:03:32.2735667Z test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest)
2023-01-11T23:03:32.2736604Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_input_moved_to_cuda_device_script>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_invalid_devices>, <__main__.TensorPipeCudaRemoteModuleTest testMethod=test_valid_device>]>
2023-01-11T23:03:32.2737639Z test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest)
2023-01-11T23:03:32.2738388Z test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest)
2023-01-11T23:03:32.2739309Z test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest)
2023-01-11T23:03:32.2740124Z test_valid_device (__main__.TensorPipeCudaRemoteModuleTest)
2023-01-11T23:03:32.2740612Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeCudaRpcTest testMethod=test_profiler_remote_cuda>]>
2023-01-11T23:03:32.2741079Z test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest)
2023-01-11T23:03:32.2742391Z <unittest.suite.TestSuite tests=[<__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_gloo_ckpt_never_find_unused>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_always>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_except_last>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never>, <__main__.TensorPipePipeWithDDPTest testMethod=test_basic_nccl_ckpt_never_find_unused>]>
2023-01-11T23:03:32.2743691Z test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest)
2023-01-11T23:03:32.2744128Z test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest)
2023-01-11T23:03:32.2744561Z test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest)
2023-01-11T23:03:32.2744977Z test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest)
2023-01-11T23:03:32.2745416Z test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest)
2023-01-11T23:03:32.2745840Z test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest)
2023-01-11T23:03:32.2746244Z test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest)
2023-01-11T23:03:32.2746678Z test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest)
2023-01-11T23:03:32.2775102Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_nested_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_async_execution_with_cuda_future>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_callback_changes_devices>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_custom_class_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_sparse_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_can_extract_list_with_cuda_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_int>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_as_str>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_device_not_cuda>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_modify_tensor_inplace>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_replace_tensor>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_cuda_future_value_on_bad_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_custom_stream_nested_multi>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_cpu_to_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_default_to_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_6>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_7>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_mixed_self_8>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_non_default_to_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_map_gpu_to_cpu_non_default>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_in_options>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_local_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_max_remote_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_invalid_min_device>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_many_to_one>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_not_timeout>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_remote_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_missing_config_response_loop>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_multi_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_one_to_many>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_remote>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_return_to_gpu_self>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_maps_wrong_worker_name>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_device_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_devices_option_mismatch_reverse>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_owner_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_as_arg_synchronization5>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_forward_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization1>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization2>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization3>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_to_here_synchronization4>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_rref_with_unpickleable_attributes>, <__main__.TensorPipeTensorPipeAgentCudaRpcTest testMethod=test_tensor_view_as_return_value>]>
2023-01-11T23:03:32.2803192Z test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2804175Z test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2805723Z test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2806728Z test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2807690Z test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2808716Z test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2809799Z test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2810939Z test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2811995Z test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2813043Z test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2813900Z test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2814778Z test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2815702Z test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2816615Z test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2817595Z test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2818521Z test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2819401Z test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2820266Z test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2821140Z test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2822022Z test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2823021Z test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2823917Z test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2824846Z test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2825778Z test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2826733Z test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2827655Z test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2828537Z test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2829410Z test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2830285Z test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2831129Z test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2832002Z test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2832811Z test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2833662Z test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2834545Z test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2835419Z test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2836333Z test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2837182Z test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2838085Z test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2838926Z test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2839787Z test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2840601Z test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2841412Z test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2842312Z test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2843208Z test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2844156Z test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2845654Z test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2846543Z test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2847527Z test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2848669Z test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2849255Z test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2849753Z test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2850227Z test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2850742Z test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2851265Z test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2851778Z test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2852310Z test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2852844Z test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2853379Z test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2853869Z test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2854360Z test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2854849Z test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2855314Z test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2855797Z test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2856298Z test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2856805Z test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2857278Z test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2857760Z test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2858261Z test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2858769Z test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2859300Z test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2859824Z test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2860346Z test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2860847Z test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2861349Z test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2861852Z test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2862334Z test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2862823Z test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2863327Z test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2863837Z test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2864329Z test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2864838Z test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2865413Z test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2865913Z test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2866416Z test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2866970Z test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2867480Z test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2867969Z test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest)
2023-01-11T23:03:32.2868892Z <unittest.suite.TestSuite tests=[<__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_device_maps_backward_pass>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_dist_autograd_sync_streams>, <__main__.TensorPipeTensorPipeCudaDistAutogradTest testMethod=test_gradients_synchronizations>]>
2023-01-11T23:03:32.2869816Z test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2023-01-11T23:03:32.2870347Z test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2023-01-11T23:03:32.2870866Z test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest)
2023-01-11T23:03:32.2871663Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2872121Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2872705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2873169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2873638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3bjzb3c6
2023-01-11T23:03:32.2874185Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3bjzb3c6/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2874493Z 
2023-01-11T23:03:32.2874606Z Running tests...
2023-01-11T23:03:32.2875006Z ----------------------------------------------------------------------
2023-01-11T23:03:32.2875590Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.2876186Z   test_ddp_dist_autograd_local_vs_remote_gpu (__main__.TensorPipeCudaDdpComparisonTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.2876708Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80274
2023-01-11T23:03:32.2877159Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80275
2023-01-11T23:03:32.2877605Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 80276
2023-01-11T23:03:32.2878049Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 80277
2023-01-11T23:03:32.2878651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2879118Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2879700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2880164Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2880743Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2881192Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2881768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2882219Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2882796Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2883301Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2883874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2885083Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2885688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2886131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2886687Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2887152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2887619Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp674f5w4r
2023-01-11T23:03:32.2888163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp674f5w4r/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2888684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm9wnc0so
2023-01-11T23:03:32.2889224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm9wnc0so/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2889761Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzyrecjqd
2023-01-11T23:03:32.2890280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzyrecjqd/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2890812Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp716jdfhz
2023-01-11T23:03:32.2891342Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp716jdfhz/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2891852Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.2892311Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.2892776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.2893245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.2893627Z skip: Need at least 4 CUDA devices (4.476s)
2023-01-11T23:03:32.2893827Z 
2023-01-11T23:03:32.2894106Z ----------------------------------------------------------------------
2023-01-11T23:03:32.2894436Z Ran 1 test in 4.476s
2023-01-11T23:03:32.2894599Z 
2023-01-11T23:03:32.2894706Z OK (skipped=1)
2023-01-11T23:03:32.2894843Z 
2023-01-11T23:03:32.2894968Z Generating XML reports...
2023-01-11T23:03:32.2895662Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20230111224457.xml
2023-01-11T23:03:32.2896442Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2896901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2897462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2897937Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2898407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpteke41hn
2023-01-11T23:03:32.2898936Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpteke41hn/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2899241Z 
2023-01-11T23:03:32.2899353Z Running tests...
2023-01-11T23:03:32.2899757Z ----------------------------------------------------------------------
2023-01-11T23:03:32.2900336Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.2900871Z   test_gpu_simple (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.2901486Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80445
2023-01-11T23:03:32.2901954Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80446
2023-01-11T23:03:32.2902444Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 80447
2023-01-11T23:03:32.2902887Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 80448
2023-01-11T23:03:32.2903507Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2903964Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2904526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2904995Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2905577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2906010Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2906586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2907055Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2907633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2908062Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2908637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2909106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2909681Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2910114Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2910688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2911155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2911606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprdjnsjto
2023-01-11T23:03:32.2912149Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprdjnsjto/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2912686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vib7ay7
2023-01-11T23:03:32.2913224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vib7ay7/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2913739Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5kjpbz97
2023-01-11T23:03:32.2914280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5kjpbz97/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2914813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7lqhr8g0
2023-01-11T23:03:32.2915332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7lqhr8g0/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2915848Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.2916323Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.2916792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.2917251Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.2917642Z fi_getinfo: -61
2023-01-11T23:03:32.2917919Z fi_getinfo: -61
2023-01-11T23:03:32.2918172Z fi_getinfo: -61
2023-01-11T23:03:32.2918440Z fi_getinfo: -61
2023-01-11T23:03:32.2918675Z ok (5.908s)
2023-01-11T23:03:32.2918822Z 
2023-01-11T23:03:32.2919124Z ----------------------------------------------------------------------
2023-01-11T23:03:32.2919458Z Ran 1 test in 5.908s
2023-01-11T23:03:32.2919619Z 
2023-01-11T23:03:32.2919756Z OK
2023-01-11T23:03:32.2919890Z 
2023-01-11T23:03:32.2920013Z Generating XML reports...
2023-01-11T23:03:32.2920678Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20230111224505.xml
2023-01-11T23:03:32.2921456Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2921910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2922471Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2922998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2923474Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5ab2aqon
2023-01-11T23:03:32.2924018Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5ab2aqon/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2924853Z 
2023-01-11T23:03:32.2924956Z Running tests...
2023-01-11T23:03:32.2925379Z ----------------------------------------------------------------------
2023-01-11T23:03:32.2925960Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.2926513Z   test_gpu_to_cpu_continuation (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.2927033Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 80868
2023-01-11T23:03:32.2927483Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 80869
2023-01-11T23:03:32.2927932Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 80870
2023-01-11T23:03:32.2928367Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 80871
2023-01-11T23:03:32.2928982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2929441Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2930018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2930475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2931050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2931498Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2932060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2932529Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2933116Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2933563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2934121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2934585Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2935162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2935589Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2936159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2936627Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2937182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvqp8ly6q
2023-01-11T23:03:32.2937722Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvqp8ly6q/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2938327Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0kz8gmkt
2023-01-11T23:03:32.2938868Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0kz8gmkt/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2939409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdcf_ae80
2023-01-11T23:03:32.2939927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdcf_ae80/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2940464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1fqvnx78
2023-01-11T23:03:32.2940993Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1fqvnx78/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2941488Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.2941969Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.2942448Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.2942928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.2943312Z fi_getinfo: -61
2023-01-11T23:03:32.2943589Z fi_getinfo: -61
2023-01-11T23:03:32.2943866Z fi_getinfo: -61
2023-01-11T23:03:32.2944123Z fi_getinfo: -61
2023-01-11T23:03:32.2944355Z ok (5.990s)
2023-01-11T23:03:32.2944501Z 
2023-01-11T23:03:32.2944772Z ----------------------------------------------------------------------
2023-01-11T23:03:32.2945087Z Ran 1 test in 5.991s
2023-01-11T23:03:32.2945249Z 
2023-01-11T23:03:32.2945358Z OK
2023-01-11T23:03:32.2945492Z 
2023-01-11T23:03:32.2945619Z Generating XML reports...
2023-01-11T23:03:32.2946311Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20230111224513.xml
2023-01-11T23:03:32.2947073Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2947534Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2948119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2948596Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2949077Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpduh4onli
2023-01-11T23:03:32.2949625Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpduh4onli/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2949931Z 
2023-01-11T23:03:32.2950043Z Running tests...
2023-01-11T23:03:32.2950436Z ----------------------------------------------------------------------
2023-01-11T23:03:32.2951026Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.2951610Z   test_gpu_to_cpu_continuation_gpu_root (__main__.TensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.2952146Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81291
2023-01-11T23:03:32.2952579Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81292
2023-01-11T23:03:32.2953026Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 81293
2023-01-11T23:03:32.2953465Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 81294
2023-01-11T23:03:32.2954049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2954514Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2955166Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2955660Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2956295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2956731Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2957302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2957770Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2958344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2958773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2959345Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2959813Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2960373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2960819Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2961390Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2961856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2962307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe6r6pdz1
2023-01-11T23:03:32.2962855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe6r6pdz1/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2963386Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ec4l56v
2023-01-11T23:03:32.2963926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ec4l56v/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2964976Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp38njxzr8
2023-01-11T23:03:32.2965529Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp38njxzr8/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2966065Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dzpgva1
2023-01-11T23:03:32.2966576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dzpgva1/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2967104Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.2967589Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.2968058Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.2968517Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.2968924Z fi_getinfo: -61
2023-01-11T23:03:32.2969203Z fi_getinfo: -61
2023-01-11T23:03:32.2969459Z fi_getinfo: -61
2023-01-11T23:03:32.2969734Z fi_getinfo: -61
2023-01-11T23:03:32.2969971Z ok (5.971s)
2023-01-11T23:03:32.2970120Z 
2023-01-11T23:03:32.2970372Z ----------------------------------------------------------------------
2023-01-11T23:03:32.2970704Z Ran 1 test in 5.971s
2023-01-11T23:03:32.2970865Z 
2023-01-11T23:03:32.2970960Z OK
2023-01-11T23:03:32.2971093Z 
2023-01-11T23:03:32.2971219Z Generating XML reports...
2023-01-11T23:03:32.2971882Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20230111224522.xml
2023-01-11T23:03:32.2972660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2973117Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2973777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2974316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2974789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp29y4yl2s
2023-01-11T23:03:32.2975334Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp29y4yl2s/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2975640Z 
2023-01-11T23:03:32.2975747Z Running tests...
2023-01-11T23:03:32.2976171Z ----------------------------------------------------------------------
2023-01-11T23:03:32.2976773Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.2977350Z   test_input_moved_to_cuda_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.2977852Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81714
2023-01-11T23:03:32.2978307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81715
2023-01-11T23:03:32.2978926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2979379Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2979935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2980403Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2980987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2981414Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2981990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2982457Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2982934Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpglvooh52
2023-01-11T23:03:32.2983468Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpglvooh52/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2984027Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdgacqoa6
2023-01-11T23:03:32.2984602Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdgacqoa6/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2985137Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.2985605Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.2986017Z fi_getinfo: -61
2023-01-11T23:03:32.2986332Z fi_getinfo: -61
2023-01-11T23:03:32.2986571Z ok (5.381s)
2023-01-11T23:03:32.2986700Z 
2023-01-11T23:03:32.2986979Z ----------------------------------------------------------------------
2023-01-11T23:03:32.2987311Z Ran 1 test in 5.382s
2023-01-11T23:03:32.2987476Z 
2023-01-11T23:03:32.2987576Z OK
2023-01-11T23:03:32.2987709Z 
2023-01-11T23:03:32.2987823Z Generating XML reports...
2023-01-11T23:03:32.2988526Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111224531.xml
2023-01-11T23:03:32.2989308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2989770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2990333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2990804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2991326Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpew76eij2
2023-01-11T23:03:32.2991879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpew76eij2/_remote_module_non_scriptable.py
2023-01-11T23:03:32.2992233Z 
2023-01-11T23:03:32.2992325Z Running tests...
2023-01-11T23:03:32.2992735Z ----------------------------------------------------------------------
2023-01-11T23:03:32.2993315Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.2993894Z   test_input_moved_to_cuda_device_script (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.2994402Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 81904
2023-01-11T23:03:32.2994851Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 81905
2023-01-11T23:03:32.2995465Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2995903Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2996481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2996957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2997538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.2997966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.2998540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.2999007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.2999457Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyt05bvx7
2023-01-11T23:03:32.3000001Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyt05bvx7/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3000539Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpx9c0rt0u
2023-01-11T23:03:32.3001083Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9c0rt0u/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3001576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3002050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3002449Z fi_getinfo: -61
2023-01-11T23:03:32.3002710Z fi_getinfo: -61
2023-01-11T23:03:32.3003212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpx9c0rt0u/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T23:03:32.3003948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyt05bvx7/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T23:03:32.3005237Z INFO:torch.distributed.nn.jit.instantiator:Skipped writing /tmp/tmpyt05bvx7/_remote_module___torch___torch_testing__internal_distributed_nn_api_remote_module_test_MyModuleInterface.py
2023-01-11T23:03:32.3005712Z ok (5.471s)
2023-01-11T23:03:32.3005860Z 
2023-01-11T23:03:32.3006143Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3006469Z Ran 1 test in 5.471s
2023-01-11T23:03:32.3006631Z 
2023-01-11T23:03:32.3006724Z OK
2023-01-11T23:03:32.3006839Z 
2023-01-11T23:03:32.3006963Z Generating XML reports...
2023-01-11T23:03:32.3007646Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111224539.xml
2023-01-11T23:03:32.3008422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3008858Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3009529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3010020Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3010551Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0t_vjzk
2023-01-11T23:03:32.3011072Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0t_vjzk/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3011373Z 
2023-01-11T23:03:32.3011482Z Running tests...
2023-01-11T23:03:32.3011896Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3012481Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3013022Z   test_invalid_devices (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3013531Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82110
2023-01-11T23:03:32.3013979Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82111
2023-01-11T23:03:32.3014579Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3015033Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3015615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3016088Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3016652Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3017104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3017680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3018134Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3018602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfex2tx0n
2023-01-11T23:03:32.3019151Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfex2tx0n/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3019686Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc3z0woca
2023-01-11T23:03:32.3020205Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc3z0woca/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3020718Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3021199Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3021593Z fi_getinfo: -61
2023-01-11T23:03:32.3021851Z fi_getinfo: -61
2023-01-11T23:03:32.3022116Z On WorkerInfo(id=1, name=worker1):
2023-01-11T23:03:32.3023093Z RuntimeError('CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n')
2023-01-11T23:03:32.3023756Z Traceback (most recent call last):
2023-01-11T23:03:32.3024279Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T23:03:32.3024737Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T23:03:32.3025319Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/nn/api/remote_module.py", line 96, in _create_module
2023-01-11T23:03:32.3025689Z     module.to(device)
2023-01-11T23:03:32.3026145Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1132, in to
2023-01-11T23:03:32.3026519Z     return self._apply(convert)
2023-01-11T23:03:32.3027062Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 807, in _apply
2023-01-11T23:03:32.3027446Z     param_applied = fn(param)
2023-01-11T23:03:32.3027933Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in convert
2023-01-11T23:03:32.3028454Z     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
2023-01-11T23:03:32.3028836Z RuntimeError: CUDA error: invalid device ordinal
2023-01-11T23:03:32.3029288Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2023-01-11T23:03:32.3029743Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2023-01-11T23:03:32.3030190Z Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2023-01-11T23:03:32.3030422Z 
2023-01-11T23:03:32.3030440Z 
2023-01-11T23:03:32.3030574Z On WorkerInfo(id=1, name=worker1):
2023-01-11T23:03:32.3033755Z RuntimeError('On WorkerInfo(id=1, name=worker1):\nRuntimeError(\'CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n\')\nTraceback (most recent call last):\n  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function\n    result = python_udf.func(*python_udf.args, **python_udf.kwargs)\n  File "/opt/conda/lib/python3.10/site-packages/torch/distributed/nn/api/remote_module.py", line 96, in _create_module\n    module.to(device)\n  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1132, in to\n    return self._apply(convert)\n  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 807, in _apply\n    param_applied = fn(param)\n  File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in convert\n    return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)\nRuntimeError: CUDA error: invalid device ordinal\nCUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.\nFor debugging consider passing CUDA_LAUNCH_BLOCKING=1.\nCompile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.\n\n')
2023-01-11T23:03:32.3035974Z Traceback (most recent call last):
2023-01-11T23:03:32.3036498Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T23:03:32.3036950Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T23:03:32.3037412Z   File "/tmp/tmpk0t_vjzk/_remote_module_non_scriptable.py", line 47, in _remote_forward
2023-01-11T23:03:32.3037764Z     module = module_rref.local_value()
2023-01-11T23:03:32.3038296Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 234, in _handle_exception
2023-01-11T23:03:32.3038665Z     raise exc
2023-01-11T23:03:32.3038941Z RuntimeError: On WorkerInfo(id=1, name=worker1):
2023-01-11T23:03:32.3039340Z RuntimeError('CUDA error: invalid device ordinal
2023-01-11T23:03:32.3039796Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2023-01-11T23:03:32.3040235Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2023-01-11T23:03:32.3040692Z Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2023-01-11T23:03:32.3041022Z ')
2023-01-11T23:03:32.3041269Z Traceback (most recent call last):
2023-01-11T23:03:32.3041769Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T23:03:32.3042221Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T23:03:32.3042799Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/nn/api/remote_module.py", line 96, in _create_module
2023-01-11T23:03:32.3043169Z     module.to(device)
2023-01-11T23:03:32.3043697Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1132, in to
2023-01-11T23:03:32.3044074Z     return self._apply(convert)
2023-01-11T23:03:32.3044896Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 807, in _apply
2023-01-11T23:03:32.3045276Z     param_applied = fn(param)
2023-01-11T23:03:32.3045760Z   File "/opt/conda/lib/python3.10/site-packages/torch/nn/modules/module.py", line 1130, in convert
2023-01-11T23:03:32.3046227Z     return t.to(device, dtype if t.is_floating_point() or t.is_complex() else None, non_blocking)
2023-01-11T23:03:32.3046610Z RuntimeError: CUDA error: invalid device ordinal
2023-01-11T23:03:32.3047067Z CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
2023-01-11T23:03:32.3047573Z For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
2023-01-11T23:03:32.3048016Z Compile with `TORCH_USE_CUDA_DSA` to enable device-side assertions.
2023-01-11T23:03:32.3048253Z 
2023-01-11T23:03:32.3048273Z 
2023-01-11T23:03:32.3048291Z 
2023-01-11T23:03:32.3048393Z ok (4.711s)
2023-01-11T23:03:32.3048540Z 
2023-01-11T23:03:32.3048815Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3049143Z Ran 1 test in 4.711s
2023-01-11T23:03:32.3049305Z 
2023-01-11T23:03:32.3049381Z OK
2023-01-11T23:03:32.3049513Z 
2023-01-11T23:03:32.3049636Z Generating XML reports...
2023-01-11T23:03:32.3050320Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111224547.xml
2023-01-11T23:03:32.3051094Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3051529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3052107Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3052586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3053038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpl69bpk00
2023-01-11T23:03:32.3053586Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpl69bpk00/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3053888Z 
2023-01-11T23:03:32.3053996Z Running tests...
2023-01-11T23:03:32.3054405Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3054963Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3055516Z   test_valid_device (__main__.TensorPipeCudaRemoteModuleTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3056018Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82299
2023-01-11T23:03:32.3056449Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82300
2023-01-11T23:03:32.3057071Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3057526Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3058109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3058569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3059153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3059607Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3060180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3060633Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3061225Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpckmv0wzb
2023-01-11T23:03:32.3061788Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpckmv0wzb/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3062384Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0l9c4wdb
2023-01-11T23:03:32.3062926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0l9c4wdb/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3063437Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3063911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3064297Z fi_getinfo: -61
2023-01-11T23:03:32.3064573Z fi_getinfo: -61
2023-01-11T23:03:32.3064803Z ok (5.399s)
2023-01-11T23:03:32.3064952Z 
2023-01-11T23:03:32.3065202Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3065529Z Ran 1 test in 5.399s
2023-01-11T23:03:32.3065689Z 
2023-01-11T23:03:32.3065788Z OK
2023-01-11T23:03:32.3065921Z 
2023-01-11T23:03:32.3066027Z Generating XML reports...
2023-01-11T23:03:32.3066704Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111224554.xml
2023-01-11T23:03:32.3067481Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3067931Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3068492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3068960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3069427Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpip95iwdb
2023-01-11T23:03:32.3069974Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpip95iwdb/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3070261Z 
2023-01-11T23:03:32.3070368Z Running tests...
2023-01-11T23:03:32.3070772Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3071350Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3071875Z   test_profiler_remote_cuda (__main__.TensorPipeCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3072358Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82489
2023-01-11T23:03:32.3072809Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82490
2023-01-11T23:03:32.3073253Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 82491
2023-01-11T23:03:32.3073679Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 82492
2023-01-11T23:03:32.3074291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3074747Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3075306Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3075784Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3076366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3076809Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3077362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3077830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3078406Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3078890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3079479Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3079998Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3080577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3081004Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3081581Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3082048Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3082512Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmezfvnp5
2023-01-11T23:03:32.3083045Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmezfvnp5/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3083581Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcb3ebr1z
2023-01-11T23:03:32.3084119Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcb3ebr1z/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3084888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphgv8yy_a
2023-01-11T23:03:32.3085430Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphgv8yy_a/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3085959Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfmrg_c8m
2023-01-11T23:03:32.3086492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfmrg_c8m/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3086983Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3087460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3087934Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3088402Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3088794Z fi_getinfo: -61
2023-01-11T23:03:32.3089070Z fi_getinfo: -61
2023-01-11T23:03:32.3089341Z fi_getinfo: -61
2023-01-11T23:03:32.3089592Z fi_getinfo: -61
2023-01-11T23:03:32.3089821Z ok (7.148s)
2023-01-11T23:03:32.3089967Z 
2023-01-11T23:03:32.3090235Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3090543Z Ran 1 test in 7.148s
2023-01-11T23:03:32.3090704Z 
2023-01-11T23:03:32.3090797Z OK
2023-01-11T23:03:32.3090929Z 
2023-01-11T23:03:32.3091054Z Generating XML reports...
2023-01-11T23:03:32.3091678Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20230111224603.xml
2023-01-11T23:03:32.3092423Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3092878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3093460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3093919Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3094388Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo9r96i3m
2023-01-11T23:03:32.3094934Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo9r96i3m/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3095234Z 
2023-01-11T23:03:32.3095345Z Running tests...
2023-01-11T23:03:32.3095735Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3096307Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3096938Z   test_basic_gloo_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3097435Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82836
2023-01-11T23:03:32.3097943Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82837
2023-01-11T23:03:32.3098564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3099017Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3099578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3100052Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3100634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3101068Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3101643Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3102114Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3102583Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpufnr3udo
2023-01-11T23:03:32.3103112Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpufnr3udo/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3103648Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeg5xopxi
2023-01-11T23:03:32.3104188Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeg5xopxi/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3104682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3105156Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3105546Z skip: Need at least 4 CUDA devices (4.305s)
2023-01-11T23:03:32.3105740Z 
2023-01-11T23:03:32.3106013Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3106323Z Ran 1 test in 4.306s
2023-01-11T23:03:32.3106485Z 
2023-01-11T23:03:32.3106593Z OK (skipped=1)
2023-01-11T23:03:32.3106747Z 
2023-01-11T23:03:32.3106869Z Generating XML reports...
2023-01-11T23:03:32.3107512Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224612.xml
2023-01-11T23:03:32.3108278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3108734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3109311Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3109771Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3110239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkwy4h9i6
2023-01-11T23:03:32.3110784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkwy4h9i6/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3111082Z 
2023-01-11T23:03:32.3111192Z Running tests...
2023-01-11T23:03:32.3111583Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3112161Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3112721Z   test_basic_gloo_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3113211Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 82939
2023-01-11T23:03:32.3113659Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 82940
2023-01-11T23:03:32.3114325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3114788Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3115401Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3115870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3116454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3116884Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3117459Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3117926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3118397Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplfil0pin
2023-01-11T23:03:32.3118929Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplfil0pin/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3119463Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0zcv7dom
2023-01-11T23:03:32.3120002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0zcv7dom/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3120509Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3120965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3121354Z skip: Need at least 4 CUDA devices (4.271s)
2023-01-11T23:03:32.3121549Z 
2023-01-11T23:03:32.3121819Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3122129Z Ran 1 test in 4.271s
2023-01-11T23:03:32.3122290Z 
2023-01-11T23:03:32.3122399Z OK (skipped=1)
2023-01-11T23:03:32.3122555Z 
2023-01-11T23:03:32.3122681Z Generating XML reports...
2023-01-11T23:03:32.3123390Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224619.xml
2023-01-11T23:03:32.3124141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3124829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3125415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3125874Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3126344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc7bojkf7
2023-01-11T23:03:32.3126890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc7bojkf7/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3127190Z 
2023-01-11T23:03:32.3127299Z Running tests...
2023-01-11T23:03:32.3127691Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3128270Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3128823Z   test_basic_gloo_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3129306Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83042
2023-01-11T23:03:32.3129755Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83043
2023-01-11T23:03:32.3130358Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3130811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3131368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3131914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3132514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3133028Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3133596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3134066Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3134534Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcfb5xzzx
2023-01-11T23:03:32.3135063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcfb5xzzx/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3135599Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf4otaju7
2023-01-11T23:03:32.3136136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf4otaju7/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3136650Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3137108Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3137501Z skip: Need at least 4 CUDA devices (4.300s)
2023-01-11T23:03:32.3137694Z 
2023-01-11T23:03:32.3137965Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3138273Z Ran 1 test in 4.300s
2023-01-11T23:03:32.3138433Z 
2023-01-11T23:03:32.3138539Z OK (skipped=1)
2023-01-11T23:03:32.3138693Z 
2023-01-11T23:03:32.3138815Z Generating XML reports...
2023-01-11T23:03:32.3139474Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224626.xml
2023-01-11T23:03:32.3140216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3140673Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3141251Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3141709Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3142177Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqpz0e78
2023-01-11T23:03:32.3142719Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqpz0e78/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3143018Z 
2023-01-11T23:03:32.3143127Z Running tests...
2023-01-11T23:03:32.3143516Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3144094Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3144661Z   test_basic_gloo_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3145179Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83145
2023-01-11T23:03:32.3145613Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83146
2023-01-11T23:03:32.3146225Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3146678Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3147245Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3147719Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3148299Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3148746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3149375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3149855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3150320Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5d43zrq0
2023-01-11T23:03:32.3150898Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5d43zrq0/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3151432Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0crnppy5
2023-01-11T23:03:32.3151971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0crnppy5/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3152487Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3152946Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3153335Z skip: Need at least 4 CUDA devices (4.390s)
2023-01-11T23:03:32.3153527Z 
2023-01-11T23:03:32.3153804Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3154129Z Ran 1 test in 4.391s
2023-01-11T23:03:32.3154274Z 
2023-01-11T23:03:32.3154380Z OK (skipped=1)
2023-01-11T23:03:32.3154536Z 
2023-01-11T23:03:32.3154660Z Generating XML reports...
2023-01-11T23:03:32.3155323Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224634.xml
2023-01-11T23:03:32.3156060Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3156510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3157087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3157563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3158018Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90cehea5
2023-01-11T23:03:32.3158565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90cehea5/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3158871Z 
2023-01-11T23:03:32.3158979Z Running tests...
2023-01-11T23:03:32.3159371Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3159951Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3160509Z   test_basic_nccl_ckpt_always (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3161015Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83248
2023-01-11T23:03:32.3161459Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83249
2023-01-11T23:03:32.3162063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3162524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3163085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3163569Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3164148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3164906Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3165474Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3165945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3166414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa6nm2yo7
2023-01-11T23:03:32.3167040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa6nm2yo7/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3167577Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7kcgc48y
2023-01-11T23:03:32.3168115Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7kcgc48y/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3168698Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3169162Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3169557Z skip: Need at least 4 CUDA devices (4.362s)
2023-01-11T23:03:32.3169758Z 
2023-01-11T23:03:32.3170030Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3170363Z Ran 1 test in 4.362s
2023-01-11T23:03:32.3170508Z 
2023-01-11T23:03:32.3170616Z OK (skipped=1)
2023-01-11T23:03:32.3170768Z 
2023-01-11T23:03:32.3170892Z Generating XML reports...
2023-01-11T23:03:32.3171555Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224641.xml
2023-01-11T23:03:32.3172295Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3172758Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3173338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3173815Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3174267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0l9ibgc
2023-01-11T23:03:32.3174812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0l9ibgc/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3175117Z 
2023-01-11T23:03:32.3175227Z Running tests...
2023-01-11T23:03:32.3175632Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3176193Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3176758Z   test_basic_nccl_ckpt_except_last (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3177274Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83351
2023-01-11T23:03:32.3177711Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83352
2023-01-11T23:03:32.3178323Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3178778Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3179365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3179821Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3180410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3180865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3181443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3181902Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3182377Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyiq2gyh0
2023-01-11T23:03:32.3182926Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyiq2gyh0/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3183448Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8eydqxl1
2023-01-11T23:03:32.3183990Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8eydqxl1/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3184505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3185037Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3185429Z skip: Need at least 4 CUDA devices (4.288s)
2023-01-11T23:03:32.3185667Z 
2023-01-11T23:03:32.3185951Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3186279Z Ran 1 test in 4.288s
2023-01-11T23:03:32.3186443Z 
2023-01-11T23:03:32.3186534Z OK (skipped=1)
2023-01-11T23:03:32.3186693Z 
2023-01-11T23:03:32.3186816Z Generating XML reports...
2023-01-11T23:03:32.3187488Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224648.xml
2023-01-11T23:03:32.3188250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3188690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3189280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3189761Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3190240Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpasg2y66e
2023-01-11T23:03:32.3190772Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpasg2y66e/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3191080Z 
2023-01-11T23:03:32.3191191Z Running tests...
2023-01-11T23:03:32.3191603Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3192159Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3192718Z   test_basic_nccl_ckpt_never (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3193226Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83454
2023-01-11T23:03:32.3193688Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83455
2023-01-11T23:03:32.3194286Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3194749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3195331Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3195812Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3196377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3196831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3197412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3197882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3198335Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jr1eq28
2023-01-11T23:03:32.3198875Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jr1eq28/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3199413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ygxwvl4
2023-01-11T23:03:32.3199950Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ygxwvl4/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3200444Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3200917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3201315Z skip: Need at least 4 CUDA devices (4.373s)
2023-01-11T23:03:32.3201507Z 
2023-01-11T23:03:32.3201782Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3202089Z Ran 1 test in 4.373s
2023-01-11T23:03:32.3202302Z 
2023-01-11T23:03:32.3202415Z OK (skipped=1)
2023-01-11T23:03:32.3202573Z 
2023-01-11T23:03:32.3202696Z Generating XML reports...
2023-01-11T23:03:32.3203340Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224655.xml
2023-01-11T23:03:32.3204154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3204843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3205435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3205898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3206364Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdz4lpem5
2023-01-11T23:03:32.3206913Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdz4lpem5/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3207215Z 
2023-01-11T23:03:32.3207305Z Running tests...
2023-01-11T23:03:32.3207709Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3208286Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3208855Z   test_basic_nccl_ckpt_never_find_unused (__main__.TensorPipePipeWithDDPTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3209349Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83557
2023-01-11T23:03:32.3209802Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83558
2023-01-11T23:03:32.3210413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3210867Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3211435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3211904Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3212490Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3212921Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3213498Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3213970Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3214436Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgm_7o6u9
2023-01-11T23:03:32.3214961Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgm_7o6u9/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3215501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa5shxy1o
2023-01-11T23:03:32.3216042Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa5shxy1o/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3216535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3217012Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3217403Z skip: Need at least 4 CUDA devices (4.289s)
2023-01-11T23:03:32.3217595Z 
2023-01-11T23:03:32.3217870Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3218179Z Ran 1 test in 4.290s
2023-01-11T23:03:32.3218337Z 
2023-01-11T23:03:32.3218451Z OK (skipped=1)
2023-01-11T23:03:32.3218602Z 
2023-01-11T23:03:32.3218724Z Generating XML reports...
2023-01-11T23:03:32.3219369Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224702.xml
2023-01-11T23:03:32.3220221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3220685Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3221327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3221788Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3222256Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcfgaxvw7
2023-01-11T23:03:32.3222803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcfgaxvw7/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3223156Z 
2023-01-11T23:03:32.3223267Z Running tests...
2023-01-11T23:03:32.3223659Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3224233Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3224850Z   test_async_execution_nested_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3225407Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 83660
2023-01-11T23:03:32.3225845Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 83661
2023-01-11T23:03:32.3226285Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 83662
2023-01-11T23:03:32.3226730Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 83663
2023-01-11T23:03:32.3227336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3227790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3228373Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3228855Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3229418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3229877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3230454Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3230910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3231486Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3231938Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3232514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3232963Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3233550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3233999Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3234575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3235031Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3235502Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1ebrkkeb
2023-01-11T23:03:32.3236047Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1ebrkkeb/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3236562Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwroyoomm
2023-01-11T23:03:32.3237095Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwroyoomm/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3237682Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7jqujr21
2023-01-11T23:03:32.3238228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7jqujr21/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3238789Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfekj278d
2023-01-11T23:03:32.3239324Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfekj278d/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3239834Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3240307Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3240763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3241235Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3241624Z fi_getinfo: -61
2023-01-11T23:03:32.3241886Z fi_getinfo: -61
2023-01-11T23:03:32.3242159Z fi_getinfo: -61
2023-01-11T23:03:32.3242430Z fi_getinfo: -61
2023-01-11T23:03:32.3242650Z ok (8.426s)
2023-01-11T23:03:32.3242793Z 
2023-01-11T23:03:32.3243066Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3243390Z Ran 1 test in 8.427s
2023-01-11T23:03:32.3243551Z 
2023-01-11T23:03:32.3243627Z OK
2023-01-11T23:03:32.3243759Z 
2023-01-11T23:03:32.3243881Z Generating XML reports...
2023-01-11T23:03:32.3244849Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224709.xml
2023-01-11T23:03:32.3245658Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3246096Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3246684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3247161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3247631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsnxtktjc
2023-01-11T23:03:32.3248158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsnxtktjc/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3248462Z 
2023-01-11T23:03:32.3248569Z Running tests...
2023-01-11T23:03:32.3248979Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3249542Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3250154Z   test_async_execution_with_cuda_future (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3250708Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84007
2023-01-11T23:03:32.3251158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84008
2023-01-11T23:03:32.3251590Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84009
2023-01-11T23:03:32.3252026Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84010
2023-01-11T23:03:32.3252646Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3253099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3253664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3254139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3254719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3255148Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3255809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3256291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3256939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3257370Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3257946Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3258416Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3258995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3259423Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3260002Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3260475Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3260929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvad7k_8y
2023-01-11T23:03:32.3261472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvad7k_8y/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3262008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfiiqgbgm
2023-01-11T23:03:32.3262545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfiiqgbgm/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3263056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7jcer_cw
2023-01-11T23:03:32.3263584Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7jcer_cw/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3264113Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptw5owmqg
2023-01-11T23:03:32.3264643Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptw5owmqg/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3265135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3265612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3266073Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3266530Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3266918Z fi_getinfo: -61
2023-01-11T23:03:32.3267203Z fi_getinfo: -61
2023-01-11T23:03:32.3267456Z fi_getinfo: -61
2023-01-11T23:03:32.3267726Z fi_getinfo: -61
2023-01-11T23:03:32.3267956Z ok (10.496s)
2023-01-11T23:03:32.3268104Z 
2023-01-11T23:03:32.3268375Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3268684Z Ran 1 test in 10.496s
2023-01-11T23:03:32.3268851Z 
2023-01-11T23:03:32.3268945Z OK
2023-01-11T23:03:32.3269076Z 
2023-01-11T23:03:32.3269199Z Generating XML reports...
2023-01-11T23:03:32.3269897Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224720.xml
2023-01-11T23:03:32.3270698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3271153Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3271729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3272189Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3272656Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vps_40y
2023-01-11T23:03:32.3273266Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vps_40y/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3273574Z 
2023-01-11T23:03:32.3273666Z Running tests...
2023-01-11T23:03:32.3274072Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3274705Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3275317Z   test_cuda_future_callback_changes_devices (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3275859Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84354
2023-01-11T23:03:32.3276309Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84355
2023-01-11T23:03:32.3276753Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84356
2023-01-11T23:03:32.3277201Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84357
2023-01-11T23:03:32.3277807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3278258Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3278842Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3279302Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3279884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3280326Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3280896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3281353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3281942Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3282391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3282944Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3283417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3284003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3284682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3285250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3285713Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3286182Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj169b4vx
2023-01-11T23:03:32.3286729Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj169b4vx/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3287242Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnngyretz
2023-01-11T23:03:32.3287789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnngyretz/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3288325Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps6zpr7wi
2023-01-11T23:03:32.3288845Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps6zpr7wi/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3289375Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcbho9duy
2023-01-11T23:03:32.3289910Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcbho9duy/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3290419Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3290961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3291439Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3291965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3292303Z ok (9.887s)
2023-01-11T23:03:32.3292435Z 
2023-01-11T23:03:32.3292712Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3293037Z Ran 1 test in 9.888s
2023-01-11T23:03:32.3293196Z 
2023-01-11T23:03:32.3293288Z OK
2023-01-11T23:03:32.3293423Z 
2023-01-11T23:03:32.3293530Z Generating XML reports...
2023-01-11T23:03:32.3294239Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224733.xml
2023-01-11T23:03:32.3295043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3295509Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3296074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3296554Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3297024Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7l8rwcmr
2023-01-11T23:03:32.3297545Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7l8rwcmr/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3297853Z 
2023-01-11T23:03:32.3297962Z Running tests...
2023-01-11T23:03:32.3298368Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3298947Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3299549Z   test_cuda_future_can_extract_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3300108Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84533
2023-01-11T23:03:32.3300561Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84534
2023-01-11T23:03:32.3301004Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84535
2023-01-11T23:03:32.3301432Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84536
2023-01-11T23:03:32.3302040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3302492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3303053Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3303525Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3304111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3304563Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3305125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3305598Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3306180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3306624Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3307184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3307651Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3308291Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3308727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3309308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3309836Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3310307Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp45bjccr_
2023-01-11T23:03:32.3310831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp45bjccr_/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3311368Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpujv4je3v
2023-01-11T23:03:32.3311906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpujv4je3v/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3312446Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphy4ncyan
2023-01-11T23:03:32.3312976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphy4ncyan/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3313513Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpek5h6vy9
2023-01-11T23:03:32.3314050Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpek5h6vy9/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3314544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3315018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3315478Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3315950Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3316280Z ok (9.268s)
2023-01-11T23:03:32.3316425Z 
2023-01-11T23:03:32.3316700Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3317032Z Ran 1 test in 9.268s
2023-01-11T23:03:32.3317192Z 
2023-01-11T23:03:32.3317268Z OK
2023-01-11T23:03:32.3317399Z 
2023-01-11T23:03:32.3317520Z Generating XML reports...
2023-01-11T23:03:32.3318227Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224746.xml
2023-01-11T23:03:32.3319023Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3319458Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3320031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3320500Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3320952Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4ucbny48
2023-01-11T23:03:32.3321494Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4ucbny48/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3321797Z 
2023-01-11T23:03:32.3321907Z Running tests...
2023-01-11T23:03:32.3322311Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3322874Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3323523Z   test_cuda_future_can_extract_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3324073Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84768
2023-01-11T23:03:32.3358824Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84769
2023-01-11T23:03:32.3359368Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84770
2023-01-11T23:03:32.3359838Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84771
2023-01-11T23:03:32.3360671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3361160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3361824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3362301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3362893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3363346Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3363912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3364721Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3365327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3365759Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3366338Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3366811Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3367399Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3367829Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3368413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3368893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3369345Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0n3zyk2v
2023-01-11T23:03:32.3369900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0n3zyk2v/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3370444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpksspfjni
2023-01-11T23:03:32.3370996Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpksspfjni/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3371522Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz3yikngg
2023-01-11T23:03:32.3372067Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz3yikngg/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3372606Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp011nozui
2023-01-11T23:03:32.3373142Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp011nozui/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3373634Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3374113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3374591Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3375054Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3375404Z ok (8.195s)
2023-01-11T23:03:32.3375555Z 
2023-01-11T23:03:32.3375836Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3376176Z Ran 1 test in 8.195s
2023-01-11T23:03:32.3376320Z 
2023-01-11T23:03:32.3376416Z OK
2023-01-11T23:03:32.3376552Z 
2023-01-11T23:03:32.3376678Z Generating XML reports...
2023-01-11T23:03:32.3377397Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224758.xml
2023-01-11T23:03:32.3378186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3378727Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3379327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3379880Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3380333Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdc0h_j_u
2023-01-11T23:03:32.3380885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdc0h_j_u/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3381192Z 
2023-01-11T23:03:32.3381304Z Running tests...
2023-01-11T23:03:32.3381698Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3382283Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3382929Z   test_cuda_future_can_extract_custom_class_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3383514Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 84943
2023-01-11T23:03:32.3383962Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 84944
2023-01-11T23:03:32.3384415Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 84945
2023-01-11T23:03:32.3384871Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 84946
2023-01-11T23:03:32.3385488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3385927Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3386510Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3386988Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3387561Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3388012Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3388591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3389061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3389623Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3390074Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3390648Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3391118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3391680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3392131Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3392712Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3393162Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3393632Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcvy9gi91
2023-01-11T23:03:32.3394186Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcvy9gi91/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3394727Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphs3527lm
2023-01-11T23:03:32.3395249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphs3527lm/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3395846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplca5sw6s
2023-01-11T23:03:32.3396403Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplca5sw6s/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3396919Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmfzaq_6e
2023-01-11T23:03:32.3397510Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmfzaq_6e/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3398024Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3398497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3398958Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3399427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3399778Z ok (9.171s)
2023-01-11T23:03:32.3399933Z 
2023-01-11T23:03:32.3400195Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3400528Z Ran 1 test in 9.171s
2023-01-11T23:03:32.3400691Z 
2023-01-11T23:03:32.3400784Z OK
2023-01-11T23:03:32.3400919Z 
2023-01-11T23:03:32.3401048Z Generating XML reports...
2023-01-11T23:03:32.3401744Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224809.xml
2023-01-11T23:03:32.3402540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3402998Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3403582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3404042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3404766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_nw6n92
2023-01-11T23:03:32.3405323Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_nw6n92/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3405624Z 
2023-01-11T23:03:32.3405717Z Running tests...
2023-01-11T23:03:32.3406142Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3406726Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3407356Z   test_cuda_future_can_extract_custom_class_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3407911Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85182
2023-01-11T23:03:32.3408373Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85183
2023-01-11T23:03:32.3408828Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85184
2023-01-11T23:03:32.3409268Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85185
2023-01-11T23:03:32.3409890Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3410352Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3410934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3411389Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3411975Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3412428Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3412988Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3413459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3414144Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3414608Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3415228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3415702Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3416282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3416728Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3417287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3417758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3418230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp860lemhj
2023-01-11T23:03:32.3418758Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp860lemhj/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3419299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn_3uz0ud
2023-01-11T23:03:32.3419844Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn_3uz0ud/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3420380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp98xscopr
2023-01-11T23:03:32.3420904Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp98xscopr/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3421444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpatp3xnk_
2023-01-11T23:03:32.3421987Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpatp3xnk_/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3422505Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3422965Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3423491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3423973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3424304Z ok (9.166s)
2023-01-11T23:03:32.3424454Z 
2023-01-11T23:03:32.3424730Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3425060Z Ran 1 test in 9.166s
2023-01-11T23:03:32.3425224Z 
2023-01-11T23:03:32.3425318Z OK
2023-01-11T23:03:32.3425434Z 
2023-01-11T23:03:32.3425558Z Generating XML reports...
2023-01-11T23:03:32.3426273Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224821.xml
2023-01-11T23:03:32.3427084Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3427522Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3428106Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3428588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3429060Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy9z2o0wi
2023-01-11T23:03:32.3429588Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy9z2o0wi/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3429892Z 
2023-01-11T23:03:32.3430003Z Running tests...
2023-01-11T23:03:32.3430412Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3430973Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3431662Z   test_cuda_future_can_extract_list_with_cuda_sparse_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3432247Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85361
2023-01-11T23:03:32.3432757Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85362
2023-01-11T23:03:32.3433192Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85363
2023-01-11T23:03:32.3433640Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85364
2023-01-11T23:03:32.3434264Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3434702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3435282Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3435762Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3436352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3436789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3437365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3437837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3438419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3438846Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3439420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3439888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3440451Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3440901Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3441485Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3441959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3442408Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp767czo84
2023-01-11T23:03:32.3442951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp767czo84/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3443489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp00ytymgj
2023-01-11T23:03:32.3444014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp00ytymgj/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3444810Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9_g2x56t
2023-01-11T23:03:32.3445351Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9_g2x56t/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3445888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvsexzkrv
2023-01-11T23:03:32.3446412Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvsexzkrv/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3446931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3447403Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3447880Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3448335Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3448679Z ok (9.280s)
2023-01-11T23:03:32.3448828Z 
2023-01-11T23:03:32.3449194Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3449520Z Ran 1 test in 9.280s
2023-01-11T23:03:32.3449683Z 
2023-01-11T23:03:32.3449780Z OK
2023-01-11T23:03:32.3449974Z 
2023-01-11T23:03:32.3450102Z Generating XML reports...
2023-01-11T23:03:32.3450824Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224833.xml
2023-01-11T23:03:32.3451604Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3452063Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3452645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3453103Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3453574Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwgsrbsrk
2023-01-11T23:03:32.3454127Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwgsrbsrk/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3454437Z 
2023-01-11T23:03:32.3454548Z Running tests...
2023-01-11T23:03:32.3454941Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3455519Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3456138Z   test_cuda_future_can_extract_list_with_cuda_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3456698Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85596
2023-01-11T23:03:32.3457136Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85597
2023-01-11T23:03:32.3457586Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85598
2023-01-11T23:03:32.3458043Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85599
2023-01-11T23:03:32.3458640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3459099Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3459680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3460161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3460727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3461176Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3461755Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3462211Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3462794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3463248Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3463827Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3464279Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3464858Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3465303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3465861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3466328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3466852Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpexy3f1xn
2023-01-11T23:03:32.3467408Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpexy3f1xn/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3467990Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf7y1k9y_
2023-01-11T23:03:32.3468538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf7y1k9y_/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3469075Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoq47qbbn
2023-01-11T23:03:32.3469621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoq47qbbn/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3470138Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3j5oevyt
2023-01-11T23:03:32.3470682Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3j5oevyt/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3471201Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3471657Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3472138Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3472608Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3472957Z ok (8.897s)
2023-01-11T23:03:32.3473087Z 
2023-01-11T23:03:32.3473365Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3473697Z Ran 1 test in 8.897s
2023-01-11T23:03:32.3473863Z 
2023-01-11T23:03:32.3473958Z OK
2023-01-11T23:03:32.3474093Z 
2023-01-11T23:03:32.3474199Z Generating XML reports...
2023-01-11T23:03:32.3474916Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224845.xml
2023-01-11T23:03:32.3475725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3476188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3476752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3477226Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3477698Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwjuhie51
2023-01-11T23:03:32.3478228Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwjuhie51/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3478533Z 
2023-01-11T23:03:32.3478645Z Running tests...
2023-01-11T23:03:32.3479051Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3479630Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3480216Z   test_cuda_future_device_as_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3480767Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85771
2023-01-11T23:03:32.3481228Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85772
2023-01-11T23:03:32.3481686Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85773
2023-01-11T23:03:32.3482120Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85774
2023-01-11T23:03:32.3482730Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3483189Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3483753Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3484534Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3485168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3485690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3486056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3486251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3486621Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3486798Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3487173Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3487367Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3487735Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3487910Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3488297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3488466Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3488728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpha5_46wr
2023-01-11T23:03:32.3489002Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpha5_46wr/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3489261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7_tnuqgy
2023-01-11T23:03:32.3489533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7_tnuqgy/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3489796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgvelne1q
2023-01-11T23:03:32.3490070Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgvelne1q/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3490329Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdyiurfv
2023-01-11T23:03:32.3490580Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdyiurfv/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3490812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3491040Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3491270Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3491499Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3491604Z ok (4.370s)
2023-01-11T23:03:32.3491624Z 
2023-01-11T23:03:32.3491904Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3492019Z Ran 1 test in 4.370s
2023-01-11T23:03:32.3492039Z 
2023-01-11T23:03:32.3492138Z OK
2023-01-11T23:03:32.3492157Z 
2023-01-11T23:03:32.3492262Z Generating XML reports...
2023-01-11T23:03:32.3492819Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224856.xml
2023-01-11T23:03:32.3493196Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3493375Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3493756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3493949Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3494260Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp82l7ukk6
2023-01-11T23:03:32.3494538Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp82l7ukk6/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3494600Z 
2023-01-11T23:03:32.3494716Z Running tests...
2023-01-11T23:03:32.3494964Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3495328Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3495638Z   test_cuda_future_device_as_int (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3495861Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 85942
2023-01-11T23:03:32.3496083Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 85943
2023-01-11T23:03:32.3496301Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 85944
2023-01-11T23:03:32.3496523Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 85945
2023-01-11T23:03:32.3496902Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3497064Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3497438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3497616Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3497999Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3498193Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3498575Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3498769Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3499140Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3499302Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3499679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3499871Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3500237Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3500412Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3500794Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3500985Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3501247Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuxe5_clo
2023-01-11T23:03:32.3501522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuxe5_clo/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3501764Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9cig61ox
2023-01-11T23:03:32.3502036Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9cig61ox/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3502290Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9ijkc49a
2023-01-11T23:03:32.3502558Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9ijkc49a/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3502813Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_57hbdit
2023-01-11T23:03:32.3503079Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_57hbdit/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3503364Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3503596Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3503868Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3504078Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3504184Z ok (4.362s)
2023-01-11T23:03:32.3504204Z 
2023-01-11T23:03:32.3504474Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3504590Z Ran 1 test in 4.362s
2023-01-11T23:03:32.3504610Z 
2023-01-11T23:03:32.3504703Z OK
2023-01-11T23:03:32.3504723Z 
2023-01-11T23:03:32.3504851Z Generating XML reports...
2023-01-11T23:03:32.3505408Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224904.xml
2023-01-11T23:03:32.3505789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3505949Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3506362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3506563Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3506829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgx3u5st2
2023-01-11T23:03:32.3507103Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgx3u5st2/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3507123Z 
2023-01-11T23:03:32.3507236Z Running tests...
2023-01-11T23:03:32.3507506Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3507870Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3508184Z   test_cuda_future_device_as_str (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3508407Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86113
2023-01-11T23:03:32.3508612Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86114
2023-01-11T23:03:32.3508832Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86115
2023-01-11T23:03:32.3509049Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86116
2023-01-11T23:03:32.3509425Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3509602Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3509986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3510183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3510555Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3510714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3511096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3511289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3511655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3511831Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3512208Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3512397Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3512821Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3513005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3513419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3513610Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3513871Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuz0ozuub
2023-01-11T23:03:32.3514150Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuz0ozuub/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3514409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpattu8xrq
2023-01-11T23:03:32.3514683Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpattu8xrq/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3514943Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcv_aytdx
2023-01-11T23:03:32.3515213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcv_aytdx/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3515467Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1szaorv6
2023-01-11T23:03:32.3515716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1szaorv6/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3515947Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3516172Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3516400Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3516630Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3516733Z ok (4.482s)
2023-01-11T23:03:32.3516753Z 
2023-01-11T23:03:32.3517029Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3517144Z Ran 1 test in 4.483s
2023-01-11T23:03:32.3517164Z 
2023-01-11T23:03:32.3517242Z OK
2023-01-11T23:03:32.3517261Z 
2023-01-11T23:03:32.3517387Z Generating XML reports...
2023-01-11T23:03:32.3517941Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224911.xml
2023-01-11T23:03:32.3518318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3518497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3518880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3519075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3519338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzeijluyf
2023-01-11T23:03:32.3519612Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzeijluyf/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3519636Z 
2023-01-11T23:03:32.3519728Z Running tests...
2023-01-11T23:03:32.3519996Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3520358Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3520672Z   test_cuda_future_device_not_cuda (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3520894Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86284
2023-01-11T23:03:32.3521112Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86285
2023-01-11T23:03:32.3521329Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86286
2023-01-11T23:03:32.3521603Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86287
2023-01-11T23:03:32.3521973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3522201Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3522592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3522786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3523202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3523385Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3523764Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3523959Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3524542Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3524714Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3525098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3525288Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3525654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3525832Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3526212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3526401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3526665Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp11she9sb
2023-01-11T23:03:32.3526918Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp11she9sb/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3527179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe7wccl4l
2023-01-11T23:03:32.3527449Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe7wccl4l/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3527707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpol71dp70
2023-01-11T23:03:32.3527976Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpol71dp70/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3528239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptfduruwr
2023-01-11T23:03:32.3528508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptfduruwr/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3528743Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3528973Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3529182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3529411Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3529513Z ok (4.388s)
2023-01-11T23:03:32.3529533Z 
2023-01-11T23:03:32.3529804Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3529918Z Ran 1 test in 4.388s
2023-01-11T23:03:32.3529939Z 
2023-01-11T23:03:32.3530033Z OK
2023-01-11T23:03:32.3530052Z 
2023-01-11T23:03:32.3530178Z Generating XML reports...
2023-01-11T23:03:32.3530732Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224918.xml
2023-01-11T23:03:32.3531167Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3531358Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3531810Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3532009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3532268Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpm20qd1zs
2023-01-11T23:03:32.3532541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpm20qd1zs/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3532560Z 
2023-01-11T23:03:32.3532671Z Running tests...
2023-01-11T23:03:32.3532935Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3533293Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3533599Z   test_cuda_future_modify_tensor_inplace (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3533822Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86455
2023-01-11T23:03:32.3534043Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86456
2023-01-11T23:03:32.3534259Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86457
2023-01-11T23:03:32.3534474Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86458
2023-01-11T23:03:32.3534854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3535031Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3535415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3535592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3535964Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3536144Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3536518Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3536714Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3537081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3537256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3537628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3537817Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3538171Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3538345Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3538719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3538907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3539168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjl9vg9s8
2023-01-11T23:03:32.3539444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjl9vg9s8/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3539702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1o1r2m06
2023-01-11T23:03:32.3539972Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1o1r2m06/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3540263Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpna44h3o1
2023-01-11T23:03:32.3540539Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpna44h3o1/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3540837Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8bjssdz
2023-01-11T23:03:32.3541105Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8bjssdz/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3541337Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3541568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3541798Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3542029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3542133Z ok (5.493s)
2023-01-11T23:03:32.3542153Z 
2023-01-11T23:03:32.3542409Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3542522Z Ran 1 test in 5.493s
2023-01-11T23:03:32.3542542Z 
2023-01-11T23:03:32.3542640Z OK
2023-01-11T23:03:32.3542659Z 
2023-01-11T23:03:32.3542784Z Generating XML reports...
2023-01-11T23:03:32.3543342Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224925.xml
2023-01-11T23:03:32.3543714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3543891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3544276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3544469Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3544710Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplyznw53f
2023-01-11T23:03:32.3544983Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplyznw53f/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3545006Z 
2023-01-11T23:03:32.3545119Z Running tests...
2023-01-11T23:03:32.3545384Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3545747Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3546061Z   test_cuda_future_replace_tensor (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3546282Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86630
2023-01-11T23:03:32.3546505Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86631
2023-01-11T23:03:32.3546702Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86632
2023-01-11T23:03:32.3546919Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86633
2023-01-11T23:03:32.3547298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3547481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3547866Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3548061Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3548430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3548605Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3548982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3549208Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3549588Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3549808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3550187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3550376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3550746Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3550920Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3551293Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3551463Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3551726Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzms_8xoy
2023-01-11T23:03:32.3551998Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzms_8xoy/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3552261Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnz7oecxv
2023-01-11T23:03:32.3552532Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnz7oecxv/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3552787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0vnaqk8x
2023-01-11T23:03:32.3553056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0vnaqk8x/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3553309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6mdf2yfm
2023-01-11T23:03:32.3553576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6mdf2yfm/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3553793Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3554022Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3554254Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3554481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3554582Z ok (5.501s)
2023-01-11T23:03:32.3554602Z 
2023-01-11T23:03:32.3554871Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3554983Z Ran 1 test in 5.501s
2023-01-11T23:03:32.3555003Z 
2023-01-11T23:03:32.3555097Z OK
2023-01-11T23:03:32.3555116Z 
2023-01-11T23:03:32.3555221Z Generating XML reports...
2023-01-11T23:03:32.3555777Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224933.xml
2023-01-11T23:03:32.3556154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3556333Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3556719Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3556910Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3557168Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa4rhuu8e
2023-01-11T23:03:32.3557436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa4rhuu8e/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3557456Z 
2023-01-11T23:03:32.3557564Z Running tests...
2023-01-11T23:03:32.3557811Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3558173Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3558579Z   test_cuda_future_value_on_bad_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3558808Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 86805
2023-01-11T23:03:32.3559067Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 86806
2023-01-11T23:03:32.3559284Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 86807
2023-01-11T23:03:32.3559498Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 86808
2023-01-11T23:03:32.3559877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3560037Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3560420Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3560613Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3560978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3561156Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3561533Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3561724Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3562085Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3562256Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3562611Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3562802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3563179Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3563351Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3563726Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3563914Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3564174Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi8bkvb51
2023-01-11T23:03:32.3564706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi8bkvb51/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3564948Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf_zzctt1
2023-01-11T23:03:32.3565219Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf_zzctt1/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3565477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxaixbx6y
2023-01-11T23:03:32.3565748Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxaixbx6y/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3566003Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpil9yh3k6
2023-01-11T23:03:32.3566267Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpil9yh3k6/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3566495Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3566725Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3566953Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3567163Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3567266Z ok (7.831s)
2023-01-11T23:03:32.3567286Z 
2023-01-11T23:03:32.3567641Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3567766Z Ran 1 test in 7.832s
2023-01-11T23:03:32.3567786Z 
2023-01-11T23:03:32.3567947Z OK
2023-01-11T23:03:32.3567967Z 
2023-01-11T23:03:32.3568094Z Generating XML reports...
2023-01-11T23:03:32.3568657Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224941.xml
2023-01-11T23:03:32.3569031Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3569210Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3569577Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3569774Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3570036Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptjmz_vl7
2023-01-11T23:03:32.3570308Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptjmz_vl7/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3570331Z 
2023-01-11T23:03:32.3570445Z Running tests...
2023-01-11T23:03:32.3570710Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3571071Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3571363Z   test_custom_stream (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3572130Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/79750 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.544s)
2023-01-11T23:03:32.3572151Z 
2023-01-11T23:03:32.3572399Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3572511Z Ran 1 test in 1.544s
2023-01-11T23:03:32.3572531Z 
2023-01-11T23:03:32.3572642Z OK (skipped=1)
2023-01-11T23:03:32.3572661Z 
2023-01-11T23:03:32.3572782Z Generating XML reports...
2023-01-11T23:03:32.3573330Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224952.xml
2023-01-11T23:03:32.3573700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3573877Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3574261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3574450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3574690Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3fsal_zo
2023-01-11T23:03:32.3574960Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3fsal_zo/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3574983Z 
2023-01-11T23:03:32.3575093Z Running tests...
2023-01-11T23:03:32.3575358Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3575720Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3576019Z   test_custom_stream_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3576240Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87018
2023-01-11T23:03:32.3576463Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87019
2023-01-11T23:03:32.3576660Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87020
2023-01-11T23:03:32.3576938Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87021
2023-01-11T23:03:32.3577328Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3577551Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3577935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3578126Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3578496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3578672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3579049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3579224Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3579596Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3579771Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3580146Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3580333Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3580698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3580874Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3581250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3581436Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3581681Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwcp81ru4
2023-01-11T23:03:32.3581951Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwcp81ru4/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3582204Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpog27ac28
2023-01-11T23:03:32.3582458Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmproa7vps9
2023-01-11T23:03:32.3582725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpog27ac28/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3582991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmproa7vps9/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3583246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu0pzz_rf
2023-01-11T23:03:32.3583512Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu0pzz_rf/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3583728Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3583956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3584187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3584416Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3584567Z fi_getinfo: -61
2023-01-11T23:03:32.3584710Z fi_getinfo: -61
2023-01-11T23:03:32.3584845Z fi_getinfo: -61
2023-01-11T23:03:32.3584983Z fi_getinfo: -61
2023-01-11T23:03:32.3585064Z ok (18.320s)
2023-01-11T23:03:32.3585084Z 
2023-01-11T23:03:32.3585351Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3585465Z Ran 1 test in 18.321s
2023-01-11T23:03:32.3585485Z 
2023-01-11T23:03:32.3585576Z OK
2023-01-11T23:03:32.3585595Z 
2023-01-11T23:03:32.3585720Z Generating XML reports...
2023-01-11T23:03:32.3586324Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224956.xml
2023-01-11T23:03:32.3586713Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3586941Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3587310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3587501Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3587758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfj6mz0og
2023-01-11T23:03:32.3588028Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfj6mz0og/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3588048Z 
2023-01-11T23:03:32.3588157Z Running tests...
2023-01-11T23:03:32.3588422Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3588787Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3589093Z   test_custom_stream_nested (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3589314Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87377
2023-01-11T23:03:32.3589516Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87378
2023-01-11T23:03:32.3589733Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87379
2023-01-11T23:03:32.3589943Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87380
2023-01-11T23:03:32.3590320Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3590499Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3590888Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3591081Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3591457Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3591612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3591991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3592184Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3592549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3592724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3593096Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3593286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3593660Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3593814Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3594184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3594373Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3594634Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnb191fw4
2023-01-11T23:03:32.3594906Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnb191fw4/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3595214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplvpvr2bj
2023-01-11T23:03:32.3595492Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplvpvr2bj/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3595747Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyqahenz
2023-01-11T23:03:32.3596063Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyqahenz/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3596299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkrvx93uc
2023-01-11T23:03:32.3596565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkrvx93uc/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3596795Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3597026Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3597256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3597486Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3597638Z fi_getinfo: -61
2023-01-11T23:03:32.3597777Z fi_getinfo: -61
2023-01-11T23:03:32.3597898Z fi_getinfo: -61
2023-01-11T23:03:32.3598035Z fi_getinfo: -61
2023-01-11T23:03:32.3598135Z ok (10.703s)
2023-01-11T23:03:32.3598155Z 
2023-01-11T23:03:32.3598423Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3598536Z Ran 1 test in 10.703s
2023-01-11T23:03:32.3598556Z 
2023-01-11T23:03:32.3598650Z OK
2023-01-11T23:03:32.3598669Z 
2023-01-11T23:03:32.3598793Z Generating XML reports...
2023-01-11T23:03:32.3599347Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225017.xml
2023-01-11T23:03:32.3599705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3599889Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3600276Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3600473Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3600732Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxfrja_5s
2023-01-11T23:03:32.3601003Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxfrja_5s/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3601023Z 
2023-01-11T23:03:32.3601133Z Running tests...
2023-01-11T23:03:32.3601396Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3601739Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3602051Z   test_custom_stream_nested_multi (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3602277Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 87736
2023-01-11T23:03:32.3602495Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 87737
2023-01-11T23:03:32.3602715Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 87738
2023-01-11T23:03:32.3602928Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 87739
2023-01-11T23:03:32.3603304Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3603481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3603851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3604008Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3604784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3604994Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3605393Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3605654Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3606026Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3606203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3606578Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3606748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3607122Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3607299Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3607675Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3607866Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3608125Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_vh7k8ny
2023-01-11T23:03:32.3608399Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_vh7k8ny/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3608661Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkgjsr2fh
2023-01-11T23:03:32.3608935Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkgjsr2fh/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3609170Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_zue8skf
2023-01-11T23:03:32.3609441Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_zue8skf/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3609696Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr20vhhkx
2023-01-11T23:03:32.3609966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr20vhhkx/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3610200Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3610434Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3610663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3610892Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3611023Z fi_getinfo: -61
2023-01-11T23:03:32.3611165Z fi_getinfo: -61
2023-01-11T23:03:32.3611302Z fi_getinfo: -61
2023-01-11T23:03:32.3611441Z fi_getinfo: -61
2023-01-11T23:03:32.3611543Z ok (9.365s)
2023-01-11T23:03:32.3611567Z 
2023-01-11T23:03:32.3611835Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3611947Z Ran 1 test in 9.365s
2023-01-11T23:03:32.3611969Z 
2023-01-11T23:03:32.3612062Z OK
2023-01-11T23:03:32.3612081Z 
2023-01-11T23:03:32.3612187Z Generating XML reports...
2023-01-11T23:03:32.3612742Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225031.xml
2023-01-11T23:03:32.3613119Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3613295Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3613677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3613873Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3614181Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnu5ke_xz
2023-01-11T23:03:32.3614464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnu5ke_xz/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3614524Z 
2023-01-11T23:03:32.3614638Z Running tests...
2023-01-11T23:03:32.3614887Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3615249Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3615546Z   test_device_map_cpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3615769Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88090
2023-01-11T23:03:32.3615989Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88091
2023-01-11T23:03:32.3616204Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88092
2023-01-11T23:03:32.3616421Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88093
2023-01-11T23:03:32.3616797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3616959Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3617346Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3617542Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3617911Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3618087Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3618461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3618657Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3619025Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3619183Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3619556Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3619743Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3620115Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3620287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3620662Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3620850Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3621118Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg2rpwt67
2023-01-11T23:03:32.3621392Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg2rpwt67/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3621633Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmph9hj1xu6
2023-01-11T23:03:32.3621905Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmph9hj1xu6/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3622162Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp9foov84
2023-01-11T23:03:32.3622431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp9foov84/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3622684Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj50uslym
2023-01-11T23:03:32.3622952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj50uslym/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3623277Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3623516Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3623769Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3623994Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3624147Z fi_getinfo: -61
2023-01-11T23:03:32.3624288Z fi_getinfo: -61
2023-01-11T23:03:32.3624426Z fi_getinfo: -61
2023-01-11T23:03:32.3624561Z fi_getinfo: -61
2023-01-11T23:03:32.3624661Z ok (4.957s)
2023-01-11T23:03:32.3624680Z 
2023-01-11T23:03:32.3624945Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3625041Z Ran 1 test in 4.957s
2023-01-11T23:03:32.3625060Z 
2023-01-11T23:03:32.3625153Z OK
2023-01-11T23:03:32.3625172Z 
2023-01-11T23:03:32.3625299Z Generating XML reports...
2023-01-11T23:03:32.3625858Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225043.xml
2023-01-11T23:03:32.3626233Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3626415Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3626798Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3626992Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3627229Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpay9_enlq
2023-01-11T23:03:32.3627499Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpay9_enlq/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3627519Z 
2023-01-11T23:03:32.3627628Z Running tests...
2023-01-11T23:03:32.3627897Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3628261Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3628582Z   test_device_map_cpu_to_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3628806Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88433
2023-01-11T23:03:32.3629026Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88434
2023-01-11T23:03:32.3629242Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88435
2023-01-11T23:03:32.3629437Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88436
2023-01-11T23:03:32.3629817Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3629993Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3630381Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3630576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3630950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3631125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3631502Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3631674Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3632043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3632218Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3632642Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3632837Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3633257Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3633433Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3633807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3633999Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3634239Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa0raqcda
2023-01-11T23:03:32.3634514Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa0raqcda/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3634774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4yb5iy1a
2023-01-11T23:03:32.3635048Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4yb5iy1a/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3635309Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjqjx7zo7
2023-01-11T23:03:32.3635578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjqjx7zo7/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3635835Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp23vbggcq
2023-01-11T23:03:32.3636101Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp23vbggcq/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3636314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3636547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3636776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3637008Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3637156Z fi_getinfo: -61
2023-01-11T23:03:32.3637303Z fi_getinfo: -61
2023-01-11T23:03:32.3637442Z fi_getinfo: -61
2023-01-11T23:03:32.3637583Z fi_getinfo: -61
2023-01-11T23:03:32.3637666Z ok (6.473s)
2023-01-11T23:03:32.3637685Z 
2023-01-11T23:03:32.3637953Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3638071Z Ran 1 test in 6.473s
2023-01-11T23:03:32.3638090Z 
2023-01-11T23:03:32.3638184Z OK
2023-01-11T23:03:32.3638202Z 
2023-01-11T23:03:32.3638328Z Generating XML reports...
2023-01-11T23:03:32.3638881Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225051.xml
2023-01-11T23:03:32.3639258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3639443Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3639809Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3640009Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3640267Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzcx6fm2x
2023-01-11T23:03:32.3640541Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzcx6fm2x/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3640561Z 
2023-01-11T23:03:32.3640669Z Running tests...
2023-01-11T23:03:32.3640936Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3641299Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3641688Z   test_device_map_cpu_to_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3641919Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 88784
2023-01-11T23:03:32.3642121Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 88785
2023-01-11T23:03:32.3642383Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 88786
2023-01-11T23:03:32.3642600Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 88787
2023-01-11T23:03:32.3642978Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3643157Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3643540Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3643734Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3644100Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3644475Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3644872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3645069Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3645435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3645611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3645985Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3646173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3646547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3646722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3647077Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3647272Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3647586Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2ra7bgbd
2023-01-11T23:03:32.3647863Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2ra7bgbd/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3648122Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps7kfgonr
2023-01-11T23:03:32.3648395Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps7kfgonr/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3648651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp58p8q5uc
2023-01-11T23:03:32.3648923Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp58p8q5uc/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3649157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_fae4y0
2023-01-11T23:03:32.3649423Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_fae4y0/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3649656Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3649887Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3650115Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3650342Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3650493Z fi_getinfo: -61
2023-01-11T23:03:32.3650635Z fi_getinfo: -61
2023-01-11T23:03:32.3650753Z fi_getinfo: -61
2023-01-11T23:03:32.3650892Z fi_getinfo: -61
2023-01-11T23:03:32.3651071Z ok (6.591s)
2023-01-11T23:03:32.3651092Z 
2023-01-11T23:03:32.3651372Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3651487Z Ran 1 test in 6.591s
2023-01-11T23:03:32.3651563Z 
2023-01-11T23:03:32.3651660Z OK
2023-01-11T23:03:32.3651679Z 
2023-01-11T23:03:32.3651807Z Generating XML reports...
2023-01-11T23:03:32.3652364Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225100.xml
2023-01-11T23:03:32.3652722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3652902Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3653288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3653483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3653744Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp78_zf4ua
2023-01-11T23:03:32.3654014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp78_zf4ua/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3654037Z 
2023-01-11T23:03:32.3654147Z Running tests...
2023-01-11T23:03:32.3654411Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3654773Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3655064Z   test_device_map_gpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3655287Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89135
2023-01-11T23:03:32.3655510Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89136
2023-01-11T23:03:32.3655731Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89137
2023-01-11T23:03:32.3655949Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89138
2023-01-11T23:03:32.3656325Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3656508Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3656893Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3657068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3657434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3657611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3657987Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3658183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3658547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3658725Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3659099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3659289Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3659645Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3659820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3660193Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3660434Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3660702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3z0ftbe
2023-01-11T23:03:32.3660979Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3z0ftbe/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3661282Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp60r6_5cu
2023-01-11T23:03:32.3661556Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp60r6_5cu/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3661791Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjq7gbd4v
2023-01-11T23:03:32.3662060Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjq7gbd4v/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3662314Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp261l0q8_
2023-01-11T23:03:32.3662578Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp261l0q8_/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3662812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3663042Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3663276Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3663506Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3663663Z fi_getinfo: -61
2023-01-11T23:03:32.3663782Z fi_getinfo: -61
2023-01-11T23:03:32.3663919Z fi_getinfo: -61
2023-01-11T23:03:32.3664056Z fi_getinfo: -61
2023-01-11T23:03:32.3664159Z ok (6.574s)
2023-01-11T23:03:32.3664178Z 
2023-01-11T23:03:32.3664443Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3664556Z Ran 1 test in 6.574s
2023-01-11T23:03:32.3664575Z 
2023-01-11T23:03:32.3664671Z OK
2023-01-11T23:03:32.3664690Z 
2023-01-11T23:03:32.3664795Z Generating XML reports...
2023-01-11T23:03:32.3665353Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225109.xml
2023-01-11T23:03:32.3665733Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3665912Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3666296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3666491Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3666746Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7b_um00c
2023-01-11T23:03:32.3667014Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7b_um00c/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3667033Z 
2023-01-11T23:03:32.3667143Z Running tests...
2023-01-11T23:03:32.3667392Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3667755Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3668087Z   test_device_map_gpu_default_to_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3668849Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/80008 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.574s)
2023-01-11T23:03:32.3668869Z 
2023-01-11T23:03:32.3669133Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3669247Z Ran 1 test in 1.575s
2023-01-11T23:03:32.3669267Z 
2023-01-11T23:03:32.3669376Z OK (skipped=1)
2023-01-11T23:03:32.3669395Z 
2023-01-11T23:03:32.3669572Z Generating XML reports...
2023-01-11T23:03:32.3670140Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225119.xml
2023-01-11T23:03:32.3670563Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3670724Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3671108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3671303Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3671564Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5y5jgeo4
2023-01-11T23:03:32.3671837Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5y5jgeo4/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3671856Z 
2023-01-11T23:03:32.3671968Z Running tests...
2023-01-11T23:03:32.3672235Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3672598Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3672887Z   test_device_map_gpu_mixed_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3673108Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89516
2023-01-11T23:03:32.3673327Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89517
2023-01-11T23:03:32.3673543Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89518
2023-01-11T23:03:32.3673755Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89519
2023-01-11T23:03:32.3674133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3674313Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3674699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3674894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3675244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3675420Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3675797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3675987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3676359Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3676533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3676909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3677099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3677448Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3677626Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3677995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3678183Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3678443Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp541rrg8a
2023-01-11T23:03:32.3678701Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwu77r87v
2023-01-11T23:03:32.3679020Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp541rrg8a/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3679301Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwu77r87v/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3679602Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8obpochj
2023-01-11T23:03:32.3679855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8obpochj/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3680111Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq8l07uzu
2023-01-11T23:03:32.3680377Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq8l07uzu/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3680612Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3680843Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3681075Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3681305Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3681458Z fi_getinfo: -61
2023-01-11T23:03:32.3681580Z fi_getinfo: -61
2023-01-11T23:03:32.3681718Z fi_getinfo: -61
2023-01-11T23:03:32.3681855Z fi_getinfo: -61
2023-01-11T23:03:32.3681959Z ok (7.764s)
2023-01-11T23:03:32.3681978Z 
2023-01-11T23:03:32.3682245Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3682359Z Ran 1 test in 7.764s
2023-01-11T23:03:32.3682378Z 
2023-01-11T23:03:32.3682472Z OK
2023-01-11T23:03:32.3682492Z 
2023-01-11T23:03:32.3682596Z Generating XML reports...
2023-01-11T23:03:32.3683151Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225123.xml
2023-01-11T23:03:32.3683529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3683706Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3684092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3684515Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3684779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_bsbtj5
2023-01-11T23:03:32.3685051Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_bsbtj5/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3685071Z 
2023-01-11T23:03:32.3685182Z Running tests...
2023-01-11T23:03:32.3685438Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3685799Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3686111Z   test_device_map_gpu_mixed_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3686334Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 89867
2023-01-11T23:03:32.3686559Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 89868
2023-01-11T23:03:32.3686776Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 89869
2023-01-11T23:03:32.3686990Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 89870
2023-01-11T23:03:32.3687368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3687546Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3687912Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3688106Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3688550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3688737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3689180Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3689375Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3689740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3689916Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3690270Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3690464Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3690843Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3691016Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3691389Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3691582Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3691846Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpai8uu34t
2023-01-11T23:03:32.3692121Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpai8uu34t/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3692378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprp3w32c3
2023-01-11T23:03:32.3692628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprp3w32c3/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3692883Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprcq8yfol
2023-01-11T23:03:32.3693155Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprcq8yfol/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3693407Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0no0m3ay
2023-01-11T23:03:32.3693676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0no0m3ay/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3693910Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3694143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3694373Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3694581Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3694730Z fi_getinfo: -61
2023-01-11T23:03:32.3694870Z fi_getinfo: -61
2023-01-11T23:03:32.3695008Z fi_getinfo: -61
2023-01-11T23:03:32.3695152Z fi_getinfo: -61
2023-01-11T23:03:32.3695254Z ok (7.675s)
2023-01-11T23:03:32.3695274Z 
2023-01-11T23:03:32.3695539Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3695636Z Ran 1 test in 7.675s
2023-01-11T23:03:32.3695675Z 
2023-01-11T23:03:32.3695749Z OK
2023-01-11T23:03:32.3695768Z 
2023-01-11T23:03:32.3695892Z Generating XML reports...
2023-01-11T23:03:32.3696449Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225133.xml
2023-01-11T23:03:32.3696822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3697001Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3697387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3697632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3697897Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6n9evf0v
2023-01-11T23:03:32.3698148Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6n9evf0v/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3698272Z 
2023-01-11T23:03:32.3698368Z Running tests...
2023-01-11T23:03:32.3698639Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3699002Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3699309Z   test_device_map_gpu_mixed_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3699534Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90218
2023-01-11T23:03:32.3699756Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90219
2023-01-11T23:03:32.3699978Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90220
2023-01-11T23:03:32.3700192Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90221
2023-01-11T23:03:32.3700549Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3700734Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3701117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3701312Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3701680Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3701857Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3702236Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3702430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3702777Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3702956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3703332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3703522Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3703892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3704066Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3704440Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3704634Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3704894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbgcg6sdi
2023-01-11T23:03:32.3705154Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbgcg6sdi/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3705413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu8m2xb3z
2023-01-11T23:03:32.3705685Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu8m2xb3z/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3705940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp587o4evb
2023-01-11T23:03:32.3706207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp587o4evb/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3706462Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbys9_mul
2023-01-11T23:03:32.3706791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbys9_mul/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3707032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3707286Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3707518Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3707746Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3707904Z fi_getinfo: -61
2023-01-11T23:03:32.3708046Z fi_getinfo: -61
2023-01-11T23:03:32.3708183Z fi_getinfo: -61
2023-01-11T23:03:32.3708320Z fi_getinfo: -61
2023-01-11T23:03:32.3708402Z ok (7.792s)
2023-01-11T23:03:32.3708443Z 
2023-01-11T23:03:32.3708690Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3708805Z Ran 1 test in 7.792s
2023-01-11T23:03:32.3708825Z 
2023-01-11T23:03:32.3708918Z OK
2023-01-11T23:03:32.3708937Z 
2023-01-11T23:03:32.3709067Z Generating XML reports...
2023-01-11T23:03:32.3709626Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225144.xml
2023-01-11T23:03:32.3710008Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3710188Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3710573Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3710748Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3711008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3tdgi39
2023-01-11T23:03:32.3711279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3tdgi39/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3711300Z 
2023-01-11T23:03:32.3711414Z Running tests...
2023-01-11T23:03:32.3711680Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3712043Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3712354Z   test_device_map_gpu_mixed_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3712578Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90569
2023-01-11T23:03:32.3712798Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90570
2023-01-11T23:03:32.3712996Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90571
2023-01-11T23:03:32.3713207Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90572
2023-01-11T23:03:32.3713584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3713767Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3714153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3714351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3714718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3714895Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3715254Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3715446Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3715815Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3716040Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3716430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3716676Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3717055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3717230Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3717606Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3717778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3718038Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5hy9wy_g
2023-01-11T23:03:32.3718317Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5hy9wy_g/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3718578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphc2xy22y
2023-01-11T23:03:32.3718851Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphc2xy22y/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3719108Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7bis5rta
2023-01-11T23:03:32.3719376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7bis5rta/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3719630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqupyu5a9
2023-01-11T23:03:32.3719879Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqupyu5a9/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3720112Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3720343Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3720576Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3720805Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3720956Z fi_getinfo: -61
2023-01-11T23:03:32.3721096Z fi_getinfo: -61
2023-01-11T23:03:32.3721233Z fi_getinfo: -61
2023-01-11T23:03:32.3721351Z fi_getinfo: -61
2023-01-11T23:03:32.3721452Z ok (7.700s)
2023-01-11T23:03:32.3721472Z 
2023-01-11T23:03:32.3721738Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3721852Z Ran 1 test in 7.700s
2023-01-11T23:03:32.3721871Z 
2023-01-11T23:03:32.3721965Z OK
2023-01-11T23:03:32.3721984Z 
2023-01-11T23:03:32.3722108Z Generating XML reports...
2023-01-11T23:03:32.3722661Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225154.xml
2023-01-11T23:03:32.3723040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3723245Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3723639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3723834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3724092Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc983gzxh
2023-01-11T23:03:32.3724576Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc983gzxh/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3724598Z 
2023-01-11T23:03:32.3724717Z Running tests...
2023-01-11T23:03:32.3724989Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3725357Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3725736Z   test_device_map_gpu_mixed_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3725948Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 90920
2023-01-11T23:03:32.3726221Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 90921
2023-01-11T23:03:32.3726439Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 90922
2023-01-11T23:03:32.3726653Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 90923
2023-01-11T23:03:32.3727036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3727216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3727600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3727794Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3728148Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3728327Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3728705Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3728898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3729265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3729440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3729813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3730004Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3730377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3730533Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3730908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3731098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3731359Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp5gtslof
2023-01-11T23:03:32.3731634Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp5gtslof/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3731893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk_pzie9v
2023-01-11T23:03:32.3732165Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk_pzie9v/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3732425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgp1v2r2f
2023-01-11T23:03:32.3732676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgp1v2r2f/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3732929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiyecc1tt
2023-01-11T23:03:32.3733199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiyecc1tt/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3733431Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3733662Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3733890Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3734117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3734265Z fi_getinfo: -61
2023-01-11T23:03:32.3734384Z fi_getinfo: -61
2023-01-11T23:03:32.3734521Z fi_getinfo: -61
2023-01-11T23:03:32.3734708Z fi_getinfo: -61
2023-01-11T23:03:32.3734818Z ok (7.693s)
2023-01-11T23:03:32.3734838Z 
2023-01-11T23:03:32.3735108Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3735275Z Ran 1 test in 7.693s
2023-01-11T23:03:32.3735295Z 
2023-01-11T23:03:32.3735389Z OK
2023-01-11T23:03:32.3735408Z 
2023-01-11T23:03:32.3735532Z Generating XML reports...
2023-01-11T23:03:32.3736074Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225205.xml
2023-01-11T23:03:32.3736447Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3736629Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3737010Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3737210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3737468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp87fup07y
2023-01-11T23:03:32.3737743Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp87fup07y/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3737763Z 
2023-01-11T23:03:32.3737874Z Running tests...
2023-01-11T23:03:32.3738122Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3738485Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3738790Z   test_device_map_gpu_mixed_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3739010Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91279
2023-01-11T23:03:32.3739230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91280
2023-01-11T23:03:32.3739453Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91281
2023-01-11T23:03:32.3739669Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 91282
2023-01-11T23:03:32.3740049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3740227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3740592Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3740786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3741153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3741328Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3741701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3741878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3742262Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3742459Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3742820Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3743011Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3743382Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3743557Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3743939Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3744182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3744452Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptlpxzeum
2023-01-11T23:03:32.3744758Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprh841qu8
2023-01-11T23:03:32.3745035Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptlpxzeum/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3745280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprh841qu8/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3745536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjr23bvb2
2023-01-11T23:03:32.3745804Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjr23bvb2/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3746058Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_yqxtkx8
2023-01-11T23:03:32.3746332Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_yqxtkx8/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3746564Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3746799Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3747032Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3747261Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3747394Z fi_getinfo: -61
2023-01-11T23:03:32.3747532Z fi_getinfo: -61
2023-01-11T23:03:32.3747670Z fi_getinfo: -61
2023-01-11T23:03:32.3747808Z fi_getinfo: -61
2023-01-11T23:03:32.3747915Z ok (7.774s)
2023-01-11T23:03:32.3747935Z 
2023-01-11T23:03:32.3748201Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3748316Z Ran 1 test in 7.774s
2023-01-11T23:03:32.3748335Z 
2023-01-11T23:03:32.3748410Z OK
2023-01-11T23:03:32.3748429Z 
2023-01-11T23:03:32.3748558Z Generating XML reports...
2023-01-11T23:03:32.3749112Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225215.xml
2023-01-11T23:03:32.3749492Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3749672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3750055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3750249Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3750507Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvajw26xv
2023-01-11T23:03:32.3750760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvajw26xv/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3750798Z 
2023-01-11T23:03:32.3750892Z Running tests...
2023-01-11T23:03:32.3751160Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3751524Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3751834Z   test_device_map_gpu_mixed_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3752060Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91638
2023-01-11T23:03:32.3752280Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91639
2023-01-11T23:03:32.3752498Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91640
2023-01-11T23:03:32.3752710Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 91641
2023-01-11T23:03:32.3753067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3753294Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3753691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3753933Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3754303Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3754481Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3754862Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3755054Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3755398Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3755576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3755950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3756142Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3756511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3756686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3757059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3757251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3757511Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpuyco0n93
2023-01-11T23:03:32.3757762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpuyco0n93/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3758023Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4df61rzk
2023-01-11T23:03:32.3758296Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4df61rzk/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3758560Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0dtd43j4
2023-01-11T23:03:32.3758829Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0dtd43j4/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3759083Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9lizlkcx
2023-01-11T23:03:32.3759349Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9lizlkcx/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3759582Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3759812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3760028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3760256Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3760408Z fi_getinfo: -61
2023-01-11T23:03:32.3760549Z fi_getinfo: -61
2023-01-11T23:03:32.3760691Z fi_getinfo: -61
2023-01-11T23:03:32.3760831Z fi_getinfo: -61
2023-01-11T23:03:32.3760933Z ok (7.694s)
2023-01-11T23:03:32.3760953Z 
2023-01-11T23:03:32.3761201Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3761315Z Ran 1 test in 7.694s
2023-01-11T23:03:32.3761334Z 
2023-01-11T23:03:32.3761427Z OK
2023-01-11T23:03:32.3761446Z 
2023-01-11T23:03:32.3761570Z Generating XML reports...
2023-01-11T23:03:32.3762123Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225226.xml
2023-01-11T23:03:32.3762550Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3762740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3763129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3763351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3763611Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0uq38rda
2023-01-11T23:03:32.3763883Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0uq38rda/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3763903Z 
2023-01-11T23:03:32.3764014Z Running tests...
2023-01-11T23:03:32.3764502Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3764883Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3765197Z   test_device_map_gpu_mixed_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3765421Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 91997
2023-01-11T23:03:32.3765645Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 91998
2023-01-11T23:03:32.3765844Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 91999
2023-01-11T23:03:32.3766059Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92000
2023-01-11T23:03:32.3766437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3766614Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3766998Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3767192Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3767564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3767740Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3768105Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3768300Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3768665Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3768843Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3769216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3769405Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3769781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3769956Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3770335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3770508Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3770767Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcl_256s0
2023-01-11T23:03:32.3771038Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcl_256s0/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3771295Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpibk3l6tu
2023-01-11T23:03:32.3771565Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpibk3l6tu/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3771911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfay_bz4j
2023-01-11T23:03:32.3772192Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfay_bz4j/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3772503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9t48wyzs
2023-01-11T23:03:32.3772767Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9t48wyzs/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3772981Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3773213Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3773441Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3773668Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3773820Z fi_getinfo: -61
2023-01-11T23:03:32.3773965Z fi_getinfo: -61
2023-01-11T23:03:32.3774103Z fi_getinfo: -61
2023-01-11T23:03:32.3774225Z fi_getinfo: -61
2023-01-11T23:03:32.3774329Z ok (7.765s)
2023-01-11T23:03:32.3774349Z 
2023-01-11T23:03:32.3774614Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3774730Z Ran 1 test in 7.765s
2023-01-11T23:03:32.3774750Z 
2023-01-11T23:03:32.3774843Z OK
2023-01-11T23:03:32.3774861Z 
2023-01-11T23:03:32.3774987Z Generating XML reports...
2023-01-11T23:03:32.3775545Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225236.xml
2023-01-11T23:03:32.3775921Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3776080Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3776463Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3776661Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3776918Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqpk1i9gf
2023-01-11T23:03:32.3777194Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqpk1i9gf/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3777213Z 
2023-01-11T23:03:32.3777324Z Running tests...
2023-01-11T23:03:32.3777593Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3777960Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3778271Z   test_device_map_gpu_mixed_self_1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3778474Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92356
2023-01-11T23:03:32.3778694Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92357
2023-01-11T23:03:32.3778914Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92358
2023-01-11T23:03:32.3779127Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92359
2023-01-11T23:03:32.3779511Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3779689Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3780075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3780269Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3780613Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3780790Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3781221Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3781422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3781791Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3782011Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3782387Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3782579Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3782949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3783103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3783476Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3783670Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3783932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiolikjee
2023-01-11T23:03:32.3784212Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiolikjee/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3784471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8uu2vy8r
2023-01-11T23:03:32.3784742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8uu2vy8r/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3784998Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfkwlyrq6
2023-01-11T23:03:32.3785249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfkwlyrq6/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3785503Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgre4kr2n
2023-01-11T23:03:32.3785773Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgre4kr2n/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3786006Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3786245Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3786475Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3786702Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3786850Z fi_getinfo: -61
2023-01-11T23:03:32.3786989Z fi_getinfo: -61
2023-01-11T23:03:32.3787109Z fi_getinfo: -61
2023-01-11T23:03:32.3787246Z fi_getinfo: -61
2023-01-11T23:03:32.3787349Z ok (7.763s)
2023-01-11T23:03:32.3787369Z 
2023-01-11T23:03:32.3787634Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3787747Z Ran 1 test in 7.763s
2023-01-11T23:03:32.3787767Z 
2023-01-11T23:03:32.3787863Z OK
2023-01-11T23:03:32.3787885Z 
2023-01-11T23:03:32.3788013Z Generating XML reports...
2023-01-11T23:03:32.3788548Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225247.xml
2023-01-11T23:03:32.3788926Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3789106Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3789493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3789686Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3789945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr_5zs2l9
2023-01-11T23:03:32.3790217Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr_5zs2l9/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3790290Z 
2023-01-11T23:03:32.3790409Z Running tests...
2023-01-11T23:03:32.3790678Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3791074Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3791388Z   test_device_map_gpu_mixed_self_2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3791610Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 92707
2023-01-11T23:03:32.3791829Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 92708
2023-01-11T23:03:32.3792048Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 92709
2023-01-11T23:03:32.3792263Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 92710
2023-01-11T23:03:32.3792639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3792820Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3793186Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3793383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3793756Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3793932Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3794309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3794503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3794868Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3795046Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3795419Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3795592Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3795965Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3796140Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3796515Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3796704Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3796966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp11tzqvpg
2023-01-11T23:03:32.3797244Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp11tzqvpg/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3797501Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpik8jwf00
2023-01-11T23:03:32.3797749Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpik8jwf00/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3798005Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpohjdmz7r
2023-01-11T23:03:32.3798275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpohjdmz7r/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3798526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqbib7otf
2023-01-11T23:03:32.3798794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqbib7otf/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3799028Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3799258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3799541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3799776Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3799952Z fi_getinfo: -61
2023-01-11T23:03:32.3800090Z fi_getinfo: -61
2023-01-11T23:03:32.3800228Z fi_getinfo: -61
2023-01-11T23:03:32.3800365Z fi_getinfo: -61
2023-01-11T23:03:32.3800467Z ok (7.776s)
2023-01-11T23:03:32.3800486Z 
2023-01-11T23:03:32.3800751Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3800864Z Ran 1 test in 7.776s
2023-01-11T23:03:32.3800883Z 
2023-01-11T23:03:32.3800958Z OK
2023-01-11T23:03:32.3800977Z 
2023-01-11T23:03:32.3801101Z Generating XML reports...
2023-01-11T23:03:32.3801660Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225257.xml
2023-01-11T23:03:32.3802040Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3802219Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3802608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3802802Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3803059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp71sgpef7
2023-01-11T23:03:32.3803329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp71sgpef7/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3803349Z 
2023-01-11T23:03:32.3803440Z Running tests...
2023-01-11T23:03:32.3803709Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3804074Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3804611Z   test_device_map_gpu_mixed_self_3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3804844Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93058
2023-01-11T23:03:32.3805068Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93059
2023-01-11T23:03:32.3805287Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93060
2023-01-11T23:03:32.3805503Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93061
2023-01-11T23:03:32.3805869Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3806047Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3806430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3806629Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3807003Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3807186Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3807564Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3807757Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3808125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3808281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3808654Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3808844Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3809289Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3809476Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3809908Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3810099Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3810362Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsyi7as9n
2023-01-11T23:03:32.3810618Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsyi7as9n/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3810875Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7xgff_rd
2023-01-11T23:03:32.3811146Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7xgff_rd/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3811404Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps2c_91zi
2023-01-11T23:03:32.3811673Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps2c_91zi/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3811932Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpav30093w
2023-01-11T23:03:32.3812196Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpav30093w/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3812429Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3812661Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3812870Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3813102Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3813252Z fi_getinfo: -61
2023-01-11T23:03:32.3813395Z fi_getinfo: -61
2023-01-11T23:03:32.3813538Z fi_getinfo: -61
2023-01-11T23:03:32.3813676Z fi_getinfo: -61
2023-01-11T23:03:32.3813778Z ok (7.700s)
2023-01-11T23:03:32.3813797Z 
2023-01-11T23:03:32.3814045Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3814161Z Ran 1 test in 7.700s
2023-01-11T23:03:32.3814181Z 
2023-01-11T23:03:32.3814274Z OK
2023-01-11T23:03:32.3814293Z 
2023-01-11T23:03:32.3814418Z Generating XML reports...
2023-01-11T23:03:32.3814973Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225308.xml
2023-01-11T23:03:32.3815350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3815529Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3815914Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3816111Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3816352Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb3ebsonw
2023-01-11T23:03:32.3816628Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb3ebsonw/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3816647Z 
2023-01-11T23:03:32.3816756Z Running tests...
2023-01-11T23:03:32.3817020Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3817383Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3817696Z   test_device_map_gpu_mixed_self_4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3817917Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93409
2023-01-11T23:03:32.3818137Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93410
2023-01-11T23:03:32.3818387Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93411
2023-01-11T23:03:32.3818609Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93412
2023-01-11T23:03:32.3819036Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3819213Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3819597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3819792Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3820160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3820337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3820717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3820890Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3821259Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3821432Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3821806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3821996Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3822363Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3822539Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3822916Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3823085Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3823391Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpobr_4kgs
2023-01-11T23:03:32.3823669Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpobr_4kgs/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3823927Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_wwlay2w
2023-01-11T23:03:32.3824198Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_wwlay2w/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3824454Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi_qqk0df
2023-01-11T23:03:32.3824725Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi_qqk0df/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3824978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9m9tmdcl
2023-01-11T23:03:32.3825251Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9m9tmdcl/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3825465Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3825700Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3825928Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3826158Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3826306Z fi_getinfo: -61
2023-01-11T23:03:32.3826446Z fi_getinfo: -61
2023-01-11T23:03:32.3826585Z fi_getinfo: -61
2023-01-11T23:03:32.3826704Z fi_getinfo: -61
2023-01-11T23:03:32.3826806Z ok (7.790s)
2023-01-11T23:03:32.3826825Z 
2023-01-11T23:03:32.3827092Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3827206Z Ran 1 test in 7.790s
2023-01-11T23:03:32.3827225Z 
2023-01-11T23:03:32.3827372Z OK
2023-01-11T23:03:32.3827392Z 
2023-01-11T23:03:32.3827525Z Generating XML reports...
2023-01-11T23:03:32.3828088Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225318.xml
2023-01-11T23:03:32.3828514Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3828692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3829056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3829252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3829508Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp69xcg5iv
2023-01-11T23:03:32.3829782Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp69xcg5iv/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3829802Z 
2023-01-11T23:03:32.3829915Z Running tests...
2023-01-11T23:03:32.3830181Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3830549Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3830863Z   test_device_map_gpu_mixed_self_5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3831063Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 93760
2023-01-11T23:03:32.3831284Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 93761
2023-01-11T23:03:32.3831501Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 93762
2023-01-11T23:03:32.3831718Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 93763
2023-01-11T23:03:32.3832098Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3832275Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3832657Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3832856Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3833223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3833380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3833758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3833951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3834318Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3834496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3834874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3835068Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3835437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3835591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3835963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3836152Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3836409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6xzl5m6w
2023-01-11T23:03:32.3836745Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6xzl5m6w/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3837013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjv9u4vxm
2023-01-11T23:03:32.3837329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjv9u4vxm/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3837585Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprzqjg8nq
2023-01-11T23:03:32.3837855Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprzqjg8nq/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3838090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptqpaufnd
2023-01-11T23:03:32.3838357Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptqpaufnd/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3838592Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3838827Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3839059Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3839288Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3839441Z fi_getinfo: -61
2023-01-11T23:03:32.3839582Z fi_getinfo: -61
2023-01-11T23:03:32.3839701Z fi_getinfo: -61
2023-01-11T23:03:32.3839840Z fi_getinfo: -61
2023-01-11T23:03:32.3839942Z ok (7.880s)
2023-01-11T23:03:32.3839962Z 
2023-01-11T23:03:32.3840229Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3840344Z Ran 1 test in 7.881s
2023-01-11T23:03:32.3840363Z 
2023-01-11T23:03:32.3840458Z OK
2023-01-11T23:03:32.3840477Z 
2023-01-11T23:03:32.3840600Z Generating XML reports...
2023-01-11T23:03:32.3841136Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225329.xml
2023-01-11T23:03:32.3841517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3841695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3842082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3842276Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3842536Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfpkvmni7
2023-01-11T23:03:32.3842812Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfpkvmni7/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3842831Z 
2023-01-11T23:03:32.3842942Z Running tests...
2023-01-11T23:03:32.3843207Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3843551Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3843870Z   test_device_map_gpu_mixed_self_6 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3844093Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94111
2023-01-11T23:03:32.3844535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94112
2023-01-11T23:03:32.3844761Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 94113
2023-01-11T23:03:32.3844976Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 94114
2023-01-11T23:03:32.3845365Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3845543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3845929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3846222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3846608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3846842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3847226Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3847422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3847785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3847960Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3848334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3848505Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3848879Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3849053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3849429Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3849618Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3849881Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxsd9sv5p
2023-01-11T23:03:32.3850157Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxsd9sv5p/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3850414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoblai9g6
2023-01-11T23:03:32.3850690Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoblai9g6/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3850929Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0zt7kt6c
2023-01-11T23:03:32.3851199Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0zt7kt6c/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3851456Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5q5p26_7
2023-01-11T23:03:32.3851724Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5q5p26_7/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3851956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3852191Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3852421Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3852651Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3852782Z fi_getinfo: -61
2023-01-11T23:03:32.3852922Z fi_getinfo: -61
2023-01-11T23:03:32.3853062Z fi_getinfo: -61
2023-01-11T23:03:32.3853198Z fi_getinfo: -61
2023-01-11T23:03:32.3853300Z ok (7.802s)
2023-01-11T23:03:32.3853320Z 
2023-01-11T23:03:32.3853589Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3853703Z Ran 1 test in 7.802s
2023-01-11T23:03:32.3853723Z 
2023-01-11T23:03:32.3853797Z OK
2023-01-11T23:03:32.3853836Z 
2023-01-11T23:03:32.3853942Z Generating XML reports...
2023-01-11T23:03:32.3854499Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225339.xml
2023-01-11T23:03:32.3854878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3855057Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3855493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3855694Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3855951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_o86yy_5
2023-01-11T23:03:32.3856262Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_o86yy_5/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3856282Z 
2023-01-11T23:03:32.3856372Z Running tests...
2023-01-11T23:03:32.3856644Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3857004Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3857322Z   test_device_map_gpu_mixed_self_7 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3857545Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94462
2023-01-11T23:03:32.3857771Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94463
2023-01-11T23:03:32.3857993Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 94464
2023-01-11T23:03:32.3858209Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 94465
2023-01-11T23:03:32.3858590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3858749Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3859121Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3859300Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3859684Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3859876Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3860258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3860450Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3860824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3860980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3861350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3861541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3861909Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3862085Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3862462Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3862652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3862912Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0hzjo05h
2023-01-11T23:03:32.3863186Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0hzjo05h/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3863423Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppj4zexjv
2023-01-11T23:03:32.3863695Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppj4zexjv/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3863951Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpttypzclr
2023-01-11T23:03:32.3864224Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpttypzclr/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3864540Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr7hmv7lg
2023-01-11T23:03:32.3864816Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr7hmv7lg/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3865050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3865331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3865543Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3865772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3865923Z fi_getinfo: -61
2023-01-11T23:03:32.3866063Z fi_getinfo: -61
2023-01-11T23:03:32.3866201Z fi_getinfo: -61
2023-01-11T23:03:32.3866340Z fi_getinfo: -61
2023-01-11T23:03:32.3866445Z ok (7.774s)
2023-01-11T23:03:32.3866465Z 
2023-01-11T23:03:32.3866730Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3866824Z Ran 1 test in 7.774s
2023-01-11T23:03:32.3866847Z 
2023-01-11T23:03:32.3866941Z OK
2023-01-11T23:03:32.3866961Z 
2023-01-11T23:03:32.3867085Z Generating XML reports...
2023-01-11T23:03:32.3867638Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225350.xml
2023-01-11T23:03:32.3868018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3868199Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3868583Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3868778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3869013Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4jn42s8q
2023-01-11T23:03:32.3869287Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4jn42s8q/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3869306Z 
2023-01-11T23:03:32.3869416Z Running tests...
2023-01-11T23:03:32.3869681Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3870049Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3870361Z   test_device_map_gpu_mixed_self_8 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3870583Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 94813
2023-01-11T23:03:32.3870803Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 94814
2023-01-11T23:03:32.3871020Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 94815
2023-01-11T23:03:32.3871215Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 94816
2023-01-11T23:03:32.3871595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3871772Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3872157Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3872351Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3872716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3872892Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3873268Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3873441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3873855Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3874036Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3874415Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3874650Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3875022Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3875197Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3875569Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3875758Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3876001Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzq0ri677
2023-01-11T23:03:32.3876275Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzq0ri677/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3876532Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmps9zy06cj
2023-01-11T23:03:32.3876805Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmps9zy06cj/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3877059Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpso68gewk
2023-01-11T23:03:32.3877326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpso68gewk/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3877576Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp09amn3m2
2023-01-11T23:03:32.3877842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp09amn3m2/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3878055Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3878291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3878522Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3878752Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3878901Z fi_getinfo: -61
2023-01-11T23:03:32.3879042Z fi_getinfo: -61
2023-01-11T23:03:32.3879180Z fi_getinfo: -61
2023-01-11T23:03:32.3879319Z fi_getinfo: -61
2023-01-11T23:03:32.3879401Z ok (7.780s)
2023-01-11T23:03:32.3879420Z 
2023-01-11T23:03:32.3879685Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3879799Z Ran 1 test in 7.780s
2023-01-11T23:03:32.3879819Z 
2023-01-11T23:03:32.3879913Z OK
2023-01-11T23:03:32.3879932Z 
2023-01-11T23:03:32.3880056Z Generating XML reports...
2023-01-11T23:03:32.3880609Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225400.xml
2023-01-11T23:03:32.3880989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3881170Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3881537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3881730Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3881987Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjx096n0m
2023-01-11T23:03:32.3882259Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjx096n0m/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3882279Z 
2023-01-11T23:03:32.3882389Z Running tests...
2023-01-11T23:03:32.3882657Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3883071Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3883396Z   test_device_map_gpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3883668Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95164
2023-01-11T23:03:32.3883869Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95165
2023-01-11T23:03:32.3884087Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95166
2023-01-11T23:03:32.3884520Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 95167
2023-01-11T23:03:32.3884924Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3885103Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3885488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3885685Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3886055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3886214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3886594Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3886786Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3887149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3887325Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3887699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3887893Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3888265Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3888442Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3888797Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3888987Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3889248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxda844dm
2023-01-11T23:03:32.3889522Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxda844dm/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3889780Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgumrm58q
2023-01-11T23:03:32.3890053Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgumrm58q/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3890313Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkkw7ibfd
2023-01-11T23:03:32.3890582Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkkw7ibfd/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3890819Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphti0o825
2023-01-11T23:03:32.3891085Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphti0o825/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3891314Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3891544Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3891767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3891995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3892143Z fi_getinfo: -61
2023-01-11T23:03:32.3892360Z fi_getinfo: -61
2023-01-11T23:03:32.3892492Z fi_getinfo: -61
2023-01-11T23:03:32.3892632Z fi_getinfo: -61
2023-01-11T23:03:32.3892734Z ok (6.475s)
2023-01-11T23:03:32.3892805Z 
2023-01-11T23:03:32.3893083Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3893197Z Ran 1 test in 6.475s
2023-01-11T23:03:32.3893217Z 
2023-01-11T23:03:32.3893311Z OK
2023-01-11T23:03:32.3893330Z 
2023-01-11T23:03:32.3893456Z Generating XML reports...
2023-01-11T23:03:32.3894008Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225411.xml
2023-01-11T23:03:32.3894361Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3894538Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3894925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3895118Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3895378Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz4pb7ukd
2023-01-11T23:03:32.3895654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz4pb7ukd/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3895674Z 
2023-01-11T23:03:32.3895785Z Running tests...
2023-01-11T23:03:32.3896048Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3896412Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3896721Z   test_device_map_gpu_non_default_to_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3896943Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95511
2023-01-11T23:03:32.3897166Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95512
2023-01-11T23:03:32.3897384Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95513
2023-01-11T23:03:32.3897605Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 95514
2023-01-11T23:03:32.3897981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3898160Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3898545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3898718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3899091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3899266Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3899647Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3899839Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3900213Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3900391Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3900765Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3900955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3901297Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3901472Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3901915Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3902112Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3902415Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb_6anwip
2023-01-11T23:03:32.3902688Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb_6anwip/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3902945Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpw2_yhw6g
2023-01-11T23:03:32.3903213Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpw2_yhw6g/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3903451Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp718hnm7_
2023-01-11T23:03:32.3903720Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp718hnm7_/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3903978Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsh2cm424
2023-01-11T23:03:32.3904245Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsh2cm424/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3904481Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3904711Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3904941Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3905171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3905324Z fi_getinfo: -61
2023-01-11T23:03:32.3905444Z fi_getinfo: -61
2023-01-11T23:03:32.3905580Z fi_getinfo: -61
2023-01-11T23:03:32.3905717Z fi_getinfo: -61
2023-01-11T23:03:32.3905819Z ok (7.772s)
2023-01-11T23:03:32.3905839Z 
2023-01-11T23:03:32.3906103Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3906219Z Ran 1 test in 7.772s
2023-01-11T23:03:32.3906238Z 
2023-01-11T23:03:32.3906332Z OK
2023-01-11T23:03:32.3906350Z 
2023-01-11T23:03:32.3906455Z Generating XML reports...
2023-01-11T23:03:32.3907014Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225420.xml
2023-01-11T23:03:32.3907388Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3907568Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3907950Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3908144Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3908402Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvi7z1a4l
2023-01-11T23:03:32.3908675Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvi7z1a4l/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3908695Z 
2023-01-11T23:03:32.3908805Z Running tests...
2023-01-11T23:03:32.3909055Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3909417Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3909734Z   test_device_map_gpu_to_cpu_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3909956Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 95870
2023-01-11T23:03:32.3910177Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 95871
2023-01-11T23:03:32.3910395Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 95872
2023-01-11T23:03:32.3910610Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 95873
2023-01-11T23:03:32.3911039Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3911203Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3913125Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3913321Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3913694Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3913870Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3914244Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3914438Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3914807Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3914983Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3915335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3915528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3915896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3916069Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3916443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3916632Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3916893Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp53_3_3lk
2023-01-11T23:03:32.3917167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp53_3_3lk/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3917426Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpytqdrq1o
2023-01-11T23:03:32.3917684Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpytqdrq1o/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3917940Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwywi0jd6
2023-01-11T23:03:32.3918210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwywi0jd6/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3918464Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5dvm6wqe
2023-01-11T23:03:32.3918728Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5dvm6wqe/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3918961Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3919196Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3919427Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3919639Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3919790Z fi_getinfo: -61
2023-01-11T23:03:32.3919926Z fi_getinfo: -61
2023-01-11T23:03:32.3920063Z fi_getinfo: -61
2023-01-11T23:03:32.3920200Z fi_getinfo: -61
2023-01-11T23:03:32.3920301Z ok (6.602s)
2023-01-11T23:03:32.3920321Z 
2023-01-11T23:03:32.3920587Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3920682Z Ran 1 test in 6.603s
2023-01-11T23:03:32.3920720Z 
2023-01-11T23:03:32.3920795Z OK
2023-01-11T23:03:32.3920814Z 
2023-01-11T23:03:32.3920938Z Generating XML reports...
2023-01-11T23:03:32.3921542Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225431.xml
2023-01-11T23:03:32.3921927Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3922151Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3922537Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3922733Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3922993Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpplxp11pl
2023-01-11T23:03:32.3923249Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpplxp11pl/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3923315Z 
2023-01-11T23:03:32.3923429Z Running tests...
2023-01-11T23:03:32.3923695Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3924065Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3924880Z   test_device_map_gpu_to_cpu_non_default (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3925127Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96221
2023-01-11T23:03:32.3925350Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96222
2023-01-11T23:03:32.3925571Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96223
2023-01-11T23:03:32.3925763Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96224
2023-01-11T23:03:32.3926158Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3926335Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3926725Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3926920Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3927290Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3927470Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3927846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3928042Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3928385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3928562Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3928935Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3929128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3929500Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3929676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3930049Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3930238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3930477Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4_slk4fd
2023-01-11T23:03:32.3930750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4_slk4fd/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3931009Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpky4gbudj
2023-01-11T23:03:32.3931371Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpky4gbudj/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3931639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp2bduj7i4
2023-01-11T23:03:32.3931964Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp2bduj7i4/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3932218Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz96_mh92
2023-01-11T23:03:32.3932481Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz96_mh92/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3932714Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3932926Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3933159Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3933386Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3933542Z fi_getinfo: -61
2023-01-11T23:03:32.3933687Z fi_getinfo: -61
2023-01-11T23:03:32.3933824Z fi_getinfo: -61
2023-01-11T23:03:32.3933961Z fi_getinfo: -61
2023-01-11T23:03:32.3934047Z ok (6.600s)
2023-01-11T23:03:32.3934085Z 
2023-01-11T23:03:32.3934334Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3934447Z Ran 1 test in 6.601s
2023-01-11T23:03:32.3934466Z 
2023-01-11T23:03:32.3934560Z OK
2023-01-11T23:03:32.3934579Z 
2023-01-11T23:03:32.3934705Z Generating XML reports...
2023-01-11T23:03:32.3935258Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225440.xml
2023-01-11T23:03:32.3935633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3935811Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3936198Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3936372Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3936638Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyhrw1phd
2023-01-11T23:03:32.3936914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyhrw1phd/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3936934Z 
2023-01-11T23:03:32.3937046Z Running tests...
2023-01-11T23:03:32.3937310Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3937670Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3937968Z   test_device_maps_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3938188Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96572
2023-01-11T23:03:32.3938392Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96573
2023-01-11T23:03:32.3938608Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96574
2023-01-11T23:03:32.3938825Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96575
2023-01-11T23:03:32.3939202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3939380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3939760Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3939955Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3940322Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3940548Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3940919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3941155Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3941521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3941695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3942068Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3942259Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3942629Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3942807Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3943161Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3943350Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3943613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47khpiro
2023-01-11T23:03:32.3943887Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47khpiro/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3944144Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcmodsw9u
2023-01-11T23:03:32.3944416Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcmodsw9u/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3944672Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_n22q4l
2023-01-11T23:03:32.3944937Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_n22q4l/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3945193Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxqyerdug
2023-01-11T23:03:32.3945440Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxqyerdug/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3945675Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3945907Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3946135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3946362Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3946509Z fi_getinfo: -61
2023-01-11T23:03:32.3946650Z fi_getinfo: -61
2023-01-11T23:03:32.3946787Z fi_getinfo: -61
2023-01-11T23:03:32.3946903Z fi_getinfo: -61
2023-01-11T23:03:32.3947004Z ok (7.806s)
2023-01-11T23:03:32.3947024Z 
2023-01-11T23:03:32.3947293Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3947463Z Ran 1 test in 7.806s
2023-01-11T23:03:32.3947483Z 
2023-01-11T23:03:32.3947578Z OK
2023-01-11T23:03:32.3947598Z 
2023-01-11T23:03:32.3947723Z Generating XML reports...
2023-01-11T23:03:32.3948281Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225449.xml
2023-01-11T23:03:32.3948656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3948815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3949199Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3949391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3949649Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3sfu6dbm
2023-01-11T23:03:32.3949973Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3sfu6dbm/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3949995Z 
2023-01-11T23:03:32.3950109Z Running tests...
2023-01-11T23:03:32.3950421Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3950784Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3951072Z   test_device_maps_in_options (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3951293Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 96931
2023-01-11T23:03:32.3951511Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 96932
2023-01-11T23:03:32.3951727Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 96933
2023-01-11T23:03:32.3951939Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 96934
2023-01-11T23:03:32.3952319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3952496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3952884Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3953078Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3953426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3953603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3953980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3954173Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3954541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3954717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3955097Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3955287Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3955640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3955816Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3956189Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3956376Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3956643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz546_z4_
2023-01-11T23:03:32.3956914Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz546_z4_/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3957171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpntr_8afg
2023-01-11T23:03:32.3957444Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpntr_8afg/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3957702Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1450yp0u
2023-01-11T23:03:32.3957948Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1450yp0u/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3958200Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfledw_pm
2023-01-11T23:03:32.3958472Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfledw_pm/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3958706Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3958988Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3959225Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3959497Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3959646Z fi_getinfo: -61
2023-01-11T23:03:32.3959764Z fi_getinfo: -61
2023-01-11T23:03:32.3959902Z fi_getinfo: -61
2023-01-11T23:03:32.3960040Z fi_getinfo: -61
2023-01-11T23:03:32.3960144Z ok (7.778s)
2023-01-11T23:03:32.3960164Z 
2023-01-11T23:03:32.3960429Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3960542Z Ran 1 test in 7.778s
2023-01-11T23:03:32.3960561Z 
2023-01-11T23:03:32.3960655Z OK
2023-01-11T23:03:32.3960674Z 
2023-01-11T23:03:32.3960797Z Generating XML reports...
2023-01-11T23:03:32.3961336Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225500.xml
2023-01-11T23:03:32.3961709Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3961890Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3962278Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3962471Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3962728Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc9jwo5vs
2023-01-11T23:03:32.3962999Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc9jwo5vs/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3963018Z 
2023-01-11T23:03:32.3963129Z Running tests...
2023-01-11T23:03:32.3963374Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3963742Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3964070Z   test_device_maps_invalid_max_local_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3964533Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97290
2023-01-11T23:03:32.3964761Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97291
2023-01-11T23:03:32.3964979Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97292
2023-01-11T23:03:32.3965194Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 97293
2023-01-11T23:03:32.3965582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3965762Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3966132Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3966327Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3966695Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3966873Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3967250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3967443Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3967806Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3967980Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3968336Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3968619Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3969007Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3969236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3969614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3969804Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3970066Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpry0hbl1j
2023-01-11T23:03:32.3970339Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpry0hbl1j/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3970596Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6ka2j8s6
2023-01-11T23:03:32.3970849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6ka2j8s6/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3971104Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0uxx6_zk
2023-01-11T23:03:32.3971374Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0uxx6_zk/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3971631Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnvjddgvz
2023-01-11T23:03:32.3971899Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnvjddgvz/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3972135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3972367Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3972599Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3972832Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3972966Z fi_getinfo: -61
2023-01-11T23:03:32.3973107Z fi_getinfo: -61
2023-01-11T23:03:32.3973244Z fi_getinfo: -61
2023-01-11T23:03:32.3973387Z fi_getinfo: -61
2023-01-11T23:03:32.3973488Z ok (4.700s)
2023-01-11T23:03:32.3973507Z 
2023-01-11T23:03:32.3973776Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3973889Z Ran 1 test in 4.700s
2023-01-11T23:03:32.3973908Z 
2023-01-11T23:03:32.3973982Z OK
2023-01-11T23:03:32.3974001Z 
2023-01-11T23:03:32.3974126Z Generating XML reports...
2023-01-11T23:03:32.3974680Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225510.xml
2023-01-11T23:03:32.3975057Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3975236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3975624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3975827Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3976090Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptf50o0dk
2023-01-11T23:03:32.3976343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptf50o0dk/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3976382Z 
2023-01-11T23:03:32.3976472Z Running tests...
2023-01-11T23:03:32.3976737Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3977104Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3977434Z   test_device_maps_invalid_max_remote_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3977710Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97477
2023-01-11T23:03:32.3977938Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97478
2023-01-11T23:03:32.3978156Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97479
2023-01-11T23:03:32.3978411Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 97480
2023-01-11T23:03:32.3978771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3978951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3979334Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3979527Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3979892Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3980071Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3980449Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3980645Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3980989Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3981164Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3981538Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3981729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3982103Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3982281Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3982655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3982847Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3983106Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0dzdkf5z
2023-01-11T23:03:32.3983362Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0dzdkf5z/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3983621Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpih8jq2wd
2023-01-11T23:03:32.3983895Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpih8jq2wd/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3984150Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpoh5l_hd2
2023-01-11T23:03:32.3984418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpoh5l_hd2/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3984675Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5e001ezi
2023-01-11T23:03:32.3984942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5e001ezi/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3985175Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3985406Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3985618Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3985845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3985994Z fi_getinfo: -61
2023-01-11T23:03:32.3986132Z fi_getinfo: -61
2023-01-11T23:03:32.3986269Z fi_getinfo: -61
2023-01-11T23:03:32.3986405Z fi_getinfo: -61
2023-01-11T23:03:32.3986505Z ok (4.598s)
2023-01-11T23:03:32.3986525Z 
2023-01-11T23:03:32.3986826Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3986946Z Ran 1 test in 4.599s
2023-01-11T23:03:32.3986965Z 
2023-01-11T23:03:32.3987058Z OK
2023-01-11T23:03:32.3987138Z 
2023-01-11T23:03:32.3987266Z Generating XML reports...
2023-01-11T23:03:32.3987828Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225518.xml
2023-01-11T23:03:32.3988206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3988387Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3988771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3988946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3989207Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvt22i8wf
2023-01-11T23:03:32.3989477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvt22i8wf/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3989500Z 
2023-01-11T23:03:32.3989609Z Running tests...
2023-01-11T23:03:32.3989876Z ----------------------------------------------------------------------
2023-01-11T23:03:32.3990242Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.3990563Z   test_device_maps_invalid_min_device (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.3990784Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97664
2023-01-11T23:03:32.3991004Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97665
2023-01-11T23:03:32.3991202Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97666
2023-01-11T23:03:32.3991419Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 97667
2023-01-11T23:03:32.3991799Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3991982Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3992366Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3992558Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3992928Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3993104Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3993461Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3993652Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3994019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3994194Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3994571Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3994760Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3995129Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.3995306Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.3995677Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.3995846Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.3996146Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn1ik6406
2023-01-11T23:03:32.3996421Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn1ik6406/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3996715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdyhbxa0z
2023-01-11T23:03:32.3996991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdyhbxa0z/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3997246Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_wkojhvl
2023-01-11T23:03:32.3997518Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_wkojhvl/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3997774Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphrbtmjzb
2023-01-11T23:03:32.3998023Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphrbtmjzb/_remote_module_non_scriptable.py
2023-01-11T23:03:32.3998258Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.3998491Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.3998724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.3998952Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.3999102Z fi_getinfo: -61
2023-01-11T23:03:32.3999243Z fi_getinfo: -61
2023-01-11T23:03:32.3999380Z fi_getinfo: -61
2023-01-11T23:03:32.3999497Z fi_getinfo: -61
2023-01-11T23:03:32.3999599Z ok (4.575s)
2023-01-11T23:03:32.3999619Z 
2023-01-11T23:03:32.3999888Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4000002Z Ran 1 test in 4.575s
2023-01-11T23:03:32.4000021Z 
2023-01-11T23:03:32.4000114Z OK
2023-01-11T23:03:32.4000133Z 
2023-01-11T23:03:32.4000257Z Generating XML reports...
2023-01-11T23:03:32.4000816Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225525.xml
2023-01-11T23:03:32.4001192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4001356Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4001740Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4001934Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4002196Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf8jelpqj
2023-01-11T23:03:32.4002469Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf8jelpqj/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4002489Z 
2023-01-11T23:03:32.4002598Z Running tests...
2023-01-11T23:03:32.4002866Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4003232Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4003537Z   test_device_maps_many_to_one (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4003742Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 97839
2023-01-11T23:03:32.4003961Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 97840
2023-01-11T23:03:32.4004178Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 97841
2023-01-11T23:03:32.4004716Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 97842
2023-01-11T23:03:32.4005108Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4005287Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4005744Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4005945Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4006352Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4006528Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4006907Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4007098Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4007464Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4007640Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4008018Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4008210Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4008580Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4008739Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4009113Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4009301Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4009563Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyk9khfz9
2023-01-11T23:03:32.4009842Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyk9khfz9/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4010101Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3hlzf1_5
2023-01-11T23:03:32.4010378Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3hlzf1_5/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4010635Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6_1x55w9
2023-01-11T23:03:32.4010884Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6_1x55w9/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4011139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkz404btb
2023-01-11T23:03:32.4011405Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkz404btb/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4011636Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4011869Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4012099Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4012331Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4012479Z fi_getinfo: -61
2023-01-11T23:03:32.4012602Z fi_getinfo: -61
2023-01-11T23:03:32.4012747Z fi_getinfo: -61
2023-01-11T23:03:32.4012885Z fi_getinfo: -61
2023-01-11T23:03:32.4012986Z ok (4.591s)
2023-01-11T23:03:32.4013006Z 
2023-01-11T23:03:32.4013272Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4013385Z Ran 1 test in 4.591s
2023-01-11T23:03:32.4013405Z 
2023-01-11T23:03:32.4013497Z OK
2023-01-11T23:03:32.4013516Z 
2023-01-11T23:03:32.4013640Z Generating XML reports...
2023-01-11T23:03:32.4014174Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225532.xml
2023-01-11T23:03:32.4014547Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4014770Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4015168Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4015407Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4015660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7cb1ukmn
2023-01-11T23:03:32.4015942Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7cb1ukmn/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4015962Z 
2023-01-11T23:03:32.4016069Z Running tests...
2023-01-11T23:03:32.4016332Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4016690Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4017002Z   test_device_maps_missing_config (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4017210Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98026
2023-01-11T23:03:32.4017428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98027
2023-01-11T23:03:32.4017645Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98028
2023-01-11T23:03:32.4017857Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 98029
2023-01-11T23:03:32.4018234Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4018409Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4018789Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4018982Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4019347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4019507Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4019883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4020075Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4020439Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4020611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4020981Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4021168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4021535Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4021692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4022062Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4022251Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4022515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpy7dvz9oo
2023-01-11T23:03:32.4022785Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpy7dvz9oo/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4023035Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_eeri22c
2023-01-11T23:03:32.4023343Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_eeri22c/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4023600Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfjdq67nq
2023-01-11T23:03:32.4023915Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfjdq67nq/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4024155Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpgwvv6lzc
2023-01-11T23:03:32.4024418Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpgwvv6lzc/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4024687Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4024917Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4025143Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4025371Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4025519Z fi_getinfo: -61
2023-01-11T23:03:32.4025655Z fi_getinfo: -61
2023-01-11T23:03:32.4025773Z fi_getinfo: -61
2023-01-11T23:03:32.4025907Z fi_getinfo: -61
2023-01-11T23:03:32.4026005Z ok (5.907s)
2023-01-11T23:03:32.4026024Z 
2023-01-11T23:03:32.4026293Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4026403Z Ran 1 test in 5.907s
2023-01-11T23:03:32.4026423Z 
2023-01-11T23:03:32.4026516Z OK
2023-01-11T23:03:32.4026535Z 
2023-01-11T23:03:32.4026656Z Generating XML reports...
2023-01-11T23:03:32.4027208Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225540.xml
2023-01-11T23:03:32.4027566Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4027742Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4028126Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4028316Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4028578Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphjm37rd2
2023-01-11T23:03:32.4028849Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphjm37rd2/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4028871Z 
2023-01-11T23:03:32.4028977Z Running tests...
2023-01-11T23:03:32.4029239Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4029582Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4029899Z   test_device_maps_missing_config_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4030117Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98373
2023-01-11T23:03:32.4030332Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98374
2023-01-11T23:03:32.4030544Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98375
2023-01-11T23:03:32.4030759Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 98376
2023-01-11T23:03:32.4031134Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4031311Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4031688Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4031861Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4032224Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4032394Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4032766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4033012Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4033384Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4033596Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4033968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4034138Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4034506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4034676Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4035046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4035230Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4035489Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvlmxmxnw
2023-01-11T23:03:32.4035762Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvlmxmxnw/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4036021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp145rbmjr
2023-01-11T23:03:32.4036288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp145rbmjr/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4036526Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplf17wpsv
2023-01-11T23:03:32.4036794Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplf17wpsv/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4037044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpifksansk
2023-01-11T23:03:32.4037310Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpifksansk/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4037541Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4037767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4037997Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4038222Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4038352Z fi_getinfo: -61
2023-01-11T23:03:32.4038489Z fi_getinfo: -61
2023-01-11T23:03:32.4038622Z fi_getinfo: -61
2023-01-11T23:03:32.4038756Z fi_getinfo: -61
2023-01-11T23:03:32.4038855Z ok (6.107s)
2023-01-11T23:03:32.4038874Z 
2023-01-11T23:03:32.4039137Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4039246Z Ran 1 test in 6.107s
2023-01-11T23:03:32.4039266Z 
2023-01-11T23:03:32.4039356Z OK
2023-01-11T23:03:32.4039375Z 
2023-01-11T23:03:32.4039480Z Generating XML reports...
2023-01-11T23:03:32.4040039Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225548.xml
2023-01-11T23:03:32.4040412Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4040591Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4040974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4041165Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4041419Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30e4siqv
2023-01-11T23:03:32.4041687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30e4siqv/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4041707Z 
2023-01-11T23:03:32.4041814Z Running tests...
2023-01-11T23:03:32.4042102Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4042477Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4042842Z   test_device_maps_missing_config_not_timeout (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4043061Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 98720
2023-01-11T23:03:32.4043276Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 98721
2023-01-11T23:03:32.4043491Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 98722
2023-01-11T23:03:32.4043700Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 98723
2023-01-11T23:03:32.4044074Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4044457Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4044872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4045063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4045430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4045604Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4045980Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4046171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4046531Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4046686Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4047063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4047250Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4047624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4047799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4048170Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4048359Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4048620Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdnc9l0lw
2023-01-11T23:03:32.4048890Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdnc9l0lw/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4049131Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp75kth9m2
2023-01-11T23:03:32.4049397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp75kth9m2/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4049654Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpge1q0ugg
2023-01-11T23:03:32.4049920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpge1q0ugg/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4050167Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3iyn38e3
2023-01-11T23:03:32.4050429Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3iyn38e3/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4050658Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4050885Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4051113Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4051393Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4051554Z fi_getinfo: -61
2023-01-11T23:03:32.4051750Z fi_getinfo: -61
2023-01-11T23:03:32.4051885Z fi_getinfo: -61
2023-01-11T23:03:32.4052017Z fi_getinfo: -61
2023-01-11T23:03:32.4052117Z ok (5.873s)
2023-01-11T23:03:32.4052136Z 
2023-01-11T23:03:32.4052398Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4052492Z Ran 1 test in 5.873s
2023-01-11T23:03:32.4052511Z 
2023-01-11T23:03:32.4052602Z OK
2023-01-11T23:03:32.4052621Z 
2023-01-11T23:03:32.4052742Z Generating XML reports...
2023-01-11T23:03:32.4053298Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225557.xml
2023-01-11T23:03:32.4053669Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4053849Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4054228Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4054422Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4054660Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8lzm5_fy
2023-01-11T23:03:32.4054927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8lzm5_fy/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4054947Z 
2023-01-11T23:03:32.4055053Z Running tests...
2023-01-11T23:03:32.4055316Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4055676Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4055999Z   test_device_maps_missing_config_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4056221Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99067
2023-01-11T23:03:32.4056439Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99068
2023-01-11T23:03:32.4056657Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99069
2023-01-11T23:03:32.4056856Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99070
2023-01-11T23:03:32.4057229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4057404Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4057784Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4057975Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4058337Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4058510Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4058885Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4059060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4059421Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4059592Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4059962Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4060148Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4060565Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4060746Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4061117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4061353Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4061595Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsgmnb5hf
2023-01-11T23:03:32.4061867Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsgmnb5hf/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4062124Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxjpnn6_g
2023-01-11T23:03:32.4062397Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxjpnn6_g/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4062651Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0z3ff2s
2023-01-11T23:03:32.4062919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0z3ff2s/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4063169Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbz0l1wzy
2023-01-11T23:03:32.4063436Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbz0l1wzy/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4063663Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4063873Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4064100Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4064324Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4064472Z fi_getinfo: -61
2023-01-11T23:03:32.4064608Z fi_getinfo: -61
2023-01-11T23:03:32.4064743Z fi_getinfo: -61
2023-01-11T23:03:32.4064876Z fi_getinfo: -61
2023-01-11T23:03:32.4064961Z ok (5.996s)
2023-01-11T23:03:32.4064981Z 
2023-01-11T23:03:32.4065243Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4065354Z Ran 1 test in 5.996s
2023-01-11T23:03:32.4065376Z 
2023-01-11T23:03:32.4065466Z OK
2023-01-11T23:03:32.4065485Z 
2023-01-11T23:03:32.4065607Z Generating XML reports...
2023-01-11T23:03:32.4066159Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225606.xml
2023-01-11T23:03:32.4066530Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4066705Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4067069Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4067258Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4067519Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp3cy3cel
2023-01-11T23:03:32.4067789Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp3cy3cel/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4067811Z 
2023-01-11T23:03:32.4067917Z Running tests...
2023-01-11T23:03:32.4068180Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4068543Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4068878Z   test_device_maps_missing_config_remote_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4069096Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99414
2023-01-11T23:03:32.4069296Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99415
2023-01-11T23:03:32.4069563Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99416
2023-01-11T23:03:32.4069782Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99417
2023-01-11T23:03:32.4070160Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4070392Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4070776Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4070968Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4071332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4071487Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4071861Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4072053Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4072416Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4072590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4072963Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4073149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4073519Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4073690Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4074043Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4074233Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4074491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp97k1rog1
2023-01-11T23:03:32.4074765Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp97k1rog1/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4075021Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30o37tx_
2023-01-11T23:03:32.4075285Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30o37tx_/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4075537Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp38wtjf3f
2023-01-11T23:03:32.4075802Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp38wtjf3f/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4076053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc8itoo3_
2023-01-11T23:03:32.4076304Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc8itoo3_/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4076535Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4076763Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4076992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4077218Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4077364Z fi_getinfo: -61
2023-01-11T23:03:32.4077501Z fi_getinfo: -61
2023-01-11T23:03:32.4077619Z fi_getinfo: -61
2023-01-11T23:03:32.4077754Z fi_getinfo: -61
2023-01-11T23:03:32.4077855Z ok (5.880s)
2023-01-11T23:03:32.4077873Z 
2023-01-11T23:03:32.4078138Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4078250Z Ran 1 test in 5.880s
2023-01-11T23:03:32.4078270Z 
2023-01-11T23:03:32.4078362Z OK
2023-01-11T23:03:32.4078380Z 
2023-01-11T23:03:32.4078553Z Generating XML reports...
2023-01-11T23:03:32.4079120Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225615.xml
2023-01-11T23:03:32.4079526Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4079701Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4080081Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4080273Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4080528Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpu5890u6z
2023-01-11T23:03:32.4080795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpu5890u6z/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4080815Z 
2023-01-11T23:03:32.4080922Z Running tests...
2023-01-11T23:03:32.4081189Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4081547Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4081859Z   test_device_maps_missing_config_response (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4082076Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 99761
2023-01-11T23:03:32.4082295Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 99762
2023-01-11T23:03:32.4082509Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 99763
2023-01-11T23:03:32.4082717Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 99764
2023-01-11T23:03:32.4083092Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4083272Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4083656Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4083834Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4084422Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4084611Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4084991Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4085182Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4085545Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4085718Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4086095Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4086283Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4086637Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4086808Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4087178Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4087365Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4087622Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7nyxniqh
2023-01-11T23:03:32.4087894Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7nyxniqh/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4088226Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpts_kdflz
2023-01-11T23:03:32.4088501Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpts_kdflz/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4088792Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp92e9xwi5
2023-01-11T23:03:32.4089055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp92e9xwi5/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4089310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_om1lgh5
2023-01-11T23:03:32.4089571Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_om1lgh5/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4089801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4090029Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4090259Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4090485Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4090638Z fi_getinfo: -61
2023-01-11T23:03:32.4090758Z fi_getinfo: -61
2023-01-11T23:03:32.4090893Z fi_getinfo: -61
2023-01-11T23:03:32.4091028Z fi_getinfo: -61
2023-01-11T23:03:32.4091126Z ok (5.992s)
2023-01-11T23:03:32.4091148Z 
2023-01-11T23:03:32.4091414Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4091525Z Ran 1 test in 5.993s
2023-01-11T23:03:32.4091544Z 
2023-01-11T23:03:32.4091636Z OK
2023-01-11T23:03:32.4091655Z 
2023-01-11T23:03:32.4091761Z Generating XML reports...
2023-01-11T23:03:32.4092318Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225623.xml
2023-01-11T23:03:32.4092691Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4092868Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4093247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4093440Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4093694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp028ps710
2023-01-11T23:03:32.4093962Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp028ps710/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4093981Z 
2023-01-11T23:03:32.4094089Z Running tests...
2023-01-11T23:03:32.4094335Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4094697Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4095030Z   test_device_maps_missing_config_response_loop (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4095249Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100108
2023-01-11T23:03:32.4095470Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100109
2023-01-11T23:03:32.4095687Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100110
2023-01-11T23:03:32.4095898Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100111
2023-01-11T23:03:32.4096273Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4096431Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4096813Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4097005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4097432Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4097613Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4098032Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4098222Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4098585Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4098757Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4099111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4099296Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4099666Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4099837Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4100206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4100390Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4100647Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp24bd3gfv
2023-01-11T23:03:32.4100917Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp24bd3gfv/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4101171Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6gyq40pl
2023-01-11T23:03:32.4101425Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6gyq40pl/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4101680Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpezc3da6b
2023-01-11T23:03:32.4101947Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpezc3da6b/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4102197Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvyu4t78u
2023-01-11T23:03:32.4102462Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvyu4t78u/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4102690Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4102918Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4103145Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4103353Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4103498Z fi_getinfo: -61
2023-01-11T23:03:32.4103638Z fi_getinfo: -61
2023-01-11T23:03:32.4103771Z fi_getinfo: -61
2023-01-11T23:03:32.4103907Z fi_getinfo: -61
2023-01-11T23:03:32.4104005Z ok (6.068s)
2023-01-11T23:03:32.4104024Z 
2023-01-11T23:03:32.4104286Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4104383Z Ran 1 test in 6.069s
2023-01-11T23:03:32.4104418Z 
2023-01-11T23:03:32.4104494Z OK
2023-01-11T23:03:32.4104513Z 
2023-01-11T23:03:32.4104633Z Generating XML reports...
2023-01-11T23:03:32.4105187Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225632.xml
2023-01-11T23:03:32.4105560Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4105737Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4106117Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4106360Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4106624Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa6pq_tdu
2023-01-11T23:03:32.4106920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa6pq_tdu/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4106940Z 
2023-01-11T23:03:32.4107046Z Running tests...
2023-01-11T23:03:32.4107322Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4107684Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4107988Z   test_device_maps_multi_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4108207Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100455
2023-01-11T23:03:32.4108428Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100456
2023-01-11T23:03:32.4108645Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100457
2023-01-11T23:03:32.4108837Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100458
2023-01-11T23:03:32.4109215Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4109390Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4109769Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4109960Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4110321Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4110496Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4110872Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4111063Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4111409Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4111584Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4111953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4112139Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4112509Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4112682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4113056Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4113245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4113487Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt8izqlq0
2023-01-11T23:03:32.4113760Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt8izqlq0/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4114014Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90y_whyx
2023-01-11T23:03:32.4114280Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90y_whyx/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4114530Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0cuz2tbn
2023-01-11T23:03:32.4114795Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0cuz2tbn/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4115044Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4rnrdtme
2023-01-11T23:03:32.4115358Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4rnrdtme/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4115593Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4115845Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4116072Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4116294Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4116445Z fi_getinfo: -61
2023-01-11T23:03:32.4116580Z fi_getinfo: -61
2023-01-11T23:03:32.4116713Z fi_getinfo: -61
2023-01-11T23:03:32.4116846Z fi_getinfo: -61
2023-01-11T23:03:32.4116928Z ok (7.930s)
2023-01-11T23:03:32.4116963Z 
2023-01-11T23:03:32.4117211Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4117320Z Ran 1 test in 7.930s
2023-01-11T23:03:32.4117339Z 
2023-01-11T23:03:32.4117430Z OK
2023-01-11T23:03:32.4117453Z 
2023-01-11T23:03:32.4117575Z Generating XML reports...
2023-01-11T23:03:32.4118129Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225641.xml
2023-01-11T23:03:32.4118508Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4118683Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4119064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4119237Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4119491Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe638btds
2023-01-11T23:03:32.4119757Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe638btds/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4119776Z 
2023-01-11T23:03:32.4119887Z Running tests...
2023-01-11T23:03:32.4120149Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4120511Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4120824Z   test_device_maps_multi_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4121046Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 100814
2023-01-11T23:03:32.4121264Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 100815
2023-01-11T23:03:32.4121463Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 100816
2023-01-11T23:03:32.4121675Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 100817
2023-01-11T23:03:32.4122050Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4122229Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4122601Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4122779Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4123159Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4123393Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4123761Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4123951Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4124532Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4124789Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4125184Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4125432Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4125800Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4125978Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4126351Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4126521Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4126779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp72pnro_j
2023-01-11T23:03:32.4127055Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp72pnro_j/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4127310Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpfj9rf5wf
2023-01-11T23:03:32.4127577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpfj9rf5wf/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4127829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9byzo4aa
2023-01-11T23:03:32.4128096Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9byzo4aa/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4128344Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbcfuww6z
2023-01-11T23:03:32.4128592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbcfuww6z/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4128823Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4129050Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4129282Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4129507Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4129657Z fi_getinfo: -61
2023-01-11T23:03:32.4129792Z fi_getinfo: -61
2023-01-11T23:03:32.4129925Z fi_getinfo: -61
2023-01-11T23:03:32.4130042Z fi_getinfo: -61
2023-01-11T23:03:32.4130140Z ok (7.801s)
2023-01-11T23:03:32.4130159Z 
2023-01-11T23:03:32.4130422Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4130533Z Ran 1 test in 7.801s
2023-01-11T23:03:32.4130552Z 
2023-01-11T23:03:32.4130648Z OK
2023-01-11T23:03:32.4130667Z 
2023-01-11T23:03:32.4130790Z Generating XML reports...
2023-01-11T23:03:32.4131343Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225651.xml
2023-01-11T23:03:32.4131717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4131878Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4132263Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4132452Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4132706Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8s3owi39
2023-01-11T23:03:32.4132971Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8s3owi39/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4132991Z 
2023-01-11T23:03:32.4133100Z Running tests...
2023-01-11T23:03:32.4133363Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4133721Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4134116Z   test_device_maps_one_to_many (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4134328Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101165
2023-01-11T23:03:32.4134589Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101166
2023-01-11T23:03:32.4134805Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101167
2023-01-11T23:03:32.4135017Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101168
2023-01-11T23:03:32.4135396Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4135573Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4135954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4136149Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4136496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4136672Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4137048Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4137238Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4137597Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4137769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4138139Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4138328Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4138700Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4138855Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4139232Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4139420Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4139679Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn7g5nqyk
2023-01-11T23:03:32.4139952Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn7g5nqyk/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4140209Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp18ttfi0g
2023-01-11T23:03:32.4140477Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp18ttfi0g/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4140731Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqs2vfsqo
2023-01-11T23:03:32.4140980Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqs2vfsqo/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4141233Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnv1hbdvq
2023-01-11T23:03:32.4141497Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnv1hbdvq/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4141727Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4141956Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4142185Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4142409Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4142556Z fi_getinfo: -61
2023-01-11T23:03:32.4142641Z ok (4.468s)
2023-01-11T23:03:32.4142677Z 
2023-01-11T23:03:32.4142976Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4143095Z Ran 1 test in 4.468s
2023-01-11T23:03:32.4143154Z 
2023-01-11T23:03:32.4143247Z OK
2023-01-11T23:03:32.4143266Z 
2023-01-11T23:03:32.4143389Z Generating XML reports...
2023-01-11T23:03:32.4143945Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225702.xml
2023-01-11T23:03:32.4144319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4144501Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4144882Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4145056Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4145318Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbkq33k1p
2023-01-11T23:03:32.4145592Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbkq33k1p/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4145615Z 
2023-01-11T23:03:32.4145722Z Running tests...
2023-01-11T23:03:32.4145987Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4146347Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4146651Z   test_device_maps_remote (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4146872Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101337
2023-01-11T23:03:32.4147073Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101338
2023-01-11T23:03:32.4147289Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101339
2023-01-11T23:03:32.4147504Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101340
2023-01-11T23:03:32.4147880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4148060Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4148441Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4148631Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4148995Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4149167Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4149524Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4149717Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4150079Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4150253Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4150628Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4150816Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4151185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4151357Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4151727Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4151897Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4152214Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc_8lews3
2023-01-11T23:03:32.4152478Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmporto9_9f
2023-01-11T23:03:32.4152791Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc_8lews3/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4153056Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmporto9_9f/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4153311Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptmhm3au5
2023-01-11T23:03:32.4153577Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptmhm3au5/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4153829Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv1ja3y2r
2023-01-11T23:03:32.4154078Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv1ja3y2r/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4154308Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4154537Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4154767Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4154992Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4155145Z fi_getinfo: -61
2023-01-11T23:03:32.4155284Z fi_getinfo: -61
2023-01-11T23:03:32.4155419Z fi_getinfo: -61
2023-01-11T23:03:32.4155539Z fi_getinfo: -61
2023-01-11T23:03:32.4155637Z ok (7.801s)
2023-01-11T23:03:32.4155657Z 
2023-01-11T23:03:32.4155929Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4156042Z Ran 1 test in 7.801s
2023-01-11T23:03:32.4156061Z 
2023-01-11T23:03:32.4156153Z OK
2023-01-11T23:03:32.4156172Z 
2023-01-11T23:03:32.4156293Z Generating XML reports...
2023-01-11T23:03:32.4156849Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225709.xml
2023-01-11T23:03:32.4157223Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4157386Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4157768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4157957Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4158213Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwnlrzs64
2023-01-11T23:03:32.4158483Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwnlrzs64/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4158502Z 
2023-01-11T23:03:32.4158610Z Running tests...
2023-01-11T23:03:32.4158872Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4159232Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4159543Z   test_device_maps_return_to_gpu (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4159746Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101696
2023-01-11T23:03:32.4159964Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101697
2023-01-11T23:03:32.4160179Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101698
2023-01-11T23:03:32.4160389Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101699
2023-01-11T23:03:32.4160766Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4160942Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4161385Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4161586Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4161990Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4162163Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4162536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4162729Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4163090Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4163263Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4163634Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4163822Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4164426Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4164603Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4164986Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4165171Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4165431Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp27bxo9yd
2023-01-11T23:03:32.4165702Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp27bxo9yd/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4165962Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo5czj62j
2023-01-11T23:03:32.4166230Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo5czj62j/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4166488Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsutcj4ut
2023-01-11T23:03:32.4166737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsutcj4ut/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4166986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3r3r1ove
2023-01-11T23:03:32.4167247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3r3r1ove/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4167477Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4167704Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4167930Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4168157Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4168305Z skip: Need at least 4 CUDA devices (4.500s)
2023-01-11T23:03:32.4168328Z 
2023-01-11T23:03:32.4168595Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4168691Z Ran 1 test in 4.501s
2023-01-11T23:03:32.4168711Z 
2023-01-11T23:03:32.4168817Z OK (skipped=1)
2023-01-11T23:03:32.4168836Z 
2023-01-11T23:03:32.4168958Z Generating XML reports...
2023-01-11T23:03:32.4169506Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225720.xml
2023-01-11T23:03:32.4169877Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4170053Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4170516Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4170718Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4171010Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpackqqei6
2023-01-11T23:03:32.4171279Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpackqqei6/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4171300Z 
2023-01-11T23:03:32.4171406Z Running tests...
2023-01-11T23:03:32.4171673Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4172032Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4172348Z   test_device_maps_return_to_gpu_self (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4172568Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 101867
2023-01-11T23:03:32.4172792Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 101868
2023-01-11T23:03:32.4173009Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 101869
2023-01-11T23:03:32.4173206Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 101870
2023-01-11T23:03:32.4173582Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4173755Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4174133Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4174324Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4174690Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4174865Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4175238Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4175430Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4175781Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4175951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4176319Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4176506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4176874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4177045Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4177417Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4177603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4177848Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpueph5m4f
2023-01-11T23:03:32.4178118Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpueph5m4f/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4178370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp17g_czav
2023-01-11T23:03:32.4178636Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp17g_czav/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4178888Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpssclfykg
2023-01-11T23:03:32.4179159Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpssclfykg/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4179460Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpycfpucou
2023-01-11T23:03:32.4179737Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpycfpucou/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4180009Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4180221Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4180446Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4180672Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4180820Z skip: Need at least 4 CUDA devices (4.481s)
2023-01-11T23:03:32.4180840Z 
2023-01-11T23:03:32.4181110Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4181222Z Ran 1 test in 4.482s
2023-01-11T23:03:32.4181241Z 
2023-01-11T23:03:32.4181346Z OK (skipped=1)
2023-01-11T23:03:32.4181368Z 
2023-01-11T23:03:32.4181490Z Generating XML reports...
2023-01-11T23:03:32.4182027Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225727.xml
2023-01-11T23:03:32.4182402Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4182576Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4182953Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4183143Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4183400Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpqzwecs9_
2023-01-11T23:03:32.4183668Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpqzwecs9_/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4183687Z 
2023-01-11T23:03:32.4183797Z Running tests...
2023-01-11T23:03:32.4184063Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4184402Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4184719Z   test_device_maps_wrong_worker_name (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4184939Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102038
2023-01-11T23:03:32.4185158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102039
2023-01-11T23:03:32.4185372Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102040
2023-01-11T23:03:32.4185581Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102041
2023-01-11T23:03:32.4185958Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4186135Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4186496Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4186688Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4187055Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4187227Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4187590Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4187763Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4188141Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4188391Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4188768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4202221Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4202718Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4202904Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4203298Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4203494Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4203759Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpkihw0vcg
2023-01-11T23:03:32.4204027Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpkihw0vcg/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4204639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp4f87_i75
2023-01-11T23:03:32.4204927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp4f87_i75/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4205192Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj3ar3jeg
2023-01-11T23:03:32.4205464Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj3ar3jeg/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4205720Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjdcaefmv
2023-01-11T23:03:32.4205991Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjdcaefmv/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4206226Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4206460Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4206678Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4206909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4207071Z fi_getinfo: -61
2023-01-11T23:03:32.4207209Z fi_getinfo: -61
2023-01-11T23:03:32.4207333Z fi_getinfo: -61
2023-01-11T23:03:32.4207462Z fi_getinfo: -61
2023-01-11T23:03:32.4207562Z ok (4.698s)
2023-01-11T23:03:32.4207584Z 
2023-01-11T23:03:32.4207837Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4207952Z Ran 1 test in 4.698s
2023-01-11T23:03:32.4207971Z 
2023-01-11T23:03:32.4208063Z OK
2023-01-11T23:03:32.4208083Z 
2023-01-11T23:03:32.4208207Z Generating XML reports...
2023-01-11T23:03:32.4208760Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225734.xml
2023-01-11T23:03:32.4209145Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4209323Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4209715Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4209898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4210142Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxzcmgywa
2023-01-11T23:03:32.4210407Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxzcmgywa/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4210426Z 
2023-01-11T23:03:32.4210537Z Running tests...
2023-01-11T23:03:32.4210792Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4211146Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4211584Z   test_device_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4211813Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102225
2023-01-11T23:03:32.4212089Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102226
2023-01-11T23:03:32.4212292Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102227
2023-01-11T23:03:32.4212505Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102228
2023-01-11T23:03:32.4212896Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4213075Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4213460Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4213655Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4214029Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4214205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4214589Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4214767Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4215131Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4215303Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4215679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4215868Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4216242Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4216416Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4216795Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4216967Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4217231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptn6zsygo
2023-01-11T23:03:32.4217508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptn6zsygo/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4217769Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp61j0q3s_
2023-01-11T23:03:32.4218041Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp61j0q3s_/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4218302Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptiohmh95
2023-01-11T23:03:32.4218574Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptiohmh95/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4218830Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0l29f0nz
2023-01-11T23:03:32.4219093Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0l29f0nz/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4219309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4219542Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4219772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4220003Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4220150Z fi_getinfo: -61
2023-01-11T23:03:32.4220287Z fi_getinfo: -61
2023-01-11T23:03:32.4220474Z fi_getinfo: -61
2023-01-11T23:03:32.4220602Z fi_getinfo: -61
2023-01-11T23:03:32.4220738Z On WorkerInfo(id=1, name=worker1):
2023-01-11T23:03:32.4221095Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!')
2023-01-11T23:03:32.4221280Z Traceback (most recent call last):
2023-01-11T23:03:32.4221645Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T23:03:32.4221839Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T23:03:32.4222250Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus
2023-01-11T23:03:32.4222373Z     return x.cpu() + y.cuda()
2023-01-11T23:03:32.4222605Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2023-01-11T23:03:32.4222644Z 
2023-01-11T23:03:32.4222759Z On WorkerInfo(id=0, name=worker0):
2023-01-11T23:03:32.4223116Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!')
2023-01-11T23:03:32.4223250Z Traceback (most recent call last):
2023-01-11T23:03:32.4223659Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T23:03:32.4223853Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T23:03:32.4224259Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus
2023-01-11T23:03:32.4224381Z     return x.cpu() + y.cuda()
2023-01-11T23:03:32.4224630Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2023-01-11T23:03:32.4224650Z 
2023-01-11T23:03:32.4224764Z On WorkerInfo(id=3, name=worker3):
2023-01-11T23:03:32.4225119Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!')
2023-01-11T23:03:32.4225252Z Traceback (most recent call last):
2023-01-11T23:03:32.4225607Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T23:03:32.4225804Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T23:03:32.4226205Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus
2023-01-11T23:03:32.4226326Z     return x.cpu() + y.cuda()
2023-01-11T23:03:32.4226573Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2023-01-11T23:03:32.4226593Z 
2023-01-11T23:03:32.4226706Z On WorkerInfo(id=2, name=worker2):
2023-01-11T23:03:32.4227056Z RuntimeError('Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!')
2023-01-11T23:03:32.4227189Z Traceback (most recent call last):
2023-01-11T23:03:32.4227546Z   File "/opt/conda/lib/python3.10/site-packages/torch/distributed/rpc/internal.py", line 207, in _run_function
2023-01-11T23:03:32.4227736Z     result = python_udf.func(*python_udf.args, **python_udf.kwargs)
2023-01-11T23:03:32.4228142Z   File "/opt/conda/lib/python3.10/site-packages/torch/testing/_internal/distributed/rpc/rpc_test.py", line 5954, in _gpu_add_wrong_gpus
2023-01-11T23:03:32.4228263Z     return x.cpu() + y.cuda()
2023-01-11T23:03:32.4228508Z RuntimeError: Expected all tensors to be on the same device, but found at least two devices, cuda:0 and cpu!
2023-01-11T23:03:32.4228527Z 
2023-01-11T23:03:32.4228627Z ok (6.656s)
2023-01-11T23:03:32.4228647Z 
2023-01-11T23:03:32.4228897Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4229009Z Ran 1 test in 6.656s
2023-01-11T23:03:32.4229029Z 
2023-01-11T23:03:32.4229121Z OK
2023-01-11T23:03:32.4229140Z 
2023-01-11T23:03:32.4229265Z Generating XML reports...
2023-01-11T23:03:32.4229876Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225742.xml
2023-01-11T23:03:32.4230267Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4230492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4230883Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4231060Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4231321Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk0ksbbgn
2023-01-11T23:03:32.4231596Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk0ksbbgn/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4231616Z 
2023-01-11T23:03:32.4231725Z Running tests...
2023-01-11T23:03:32.4231987Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4232357Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4232669Z   test_devices_option_mismatch (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4232899Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102572
2023-01-11T23:03:32.4233122Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102573
2023-01-11T23:03:32.4233322Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102574
2023-01-11T23:03:32.4233541Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102575
2023-01-11T23:03:32.4233919Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4234097Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4234489Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4234682Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4235064Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4235240Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4235620Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4235795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4236162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4236338Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4236717Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4236908Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4237271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4237449Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4237833Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4238005Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4238266Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7ld9or14
2023-01-11T23:03:32.4238540Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7ld9or14/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4238797Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpte1l8s07
2023-01-11T23:03:32.4239134Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpte1l8s07/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4239401Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_498aea3
2023-01-11T23:03:32.4239712Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_498aea3/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4239968Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpvxvmn9f9
2023-01-11T23:03:32.4240240Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpvxvmn9f9/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4240456Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4240682Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4240911Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4241144Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4241298Z fi_getinfo: -61
2023-01-11T23:03:32.4241438Z fi_getinfo: -61
2023-01-11T23:03:32.4241574Z fi_getinfo: -61
2023-01-11T23:03:32.4241696Z fi_getinfo: -61
2023-01-11T23:03:32.4241796Z ok (4.684s)
2023-01-11T23:03:32.4241816Z 
2023-01-11T23:03:32.4242080Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4242191Z Ran 1 test in 4.684s
2023-01-11T23:03:32.4242210Z 
2023-01-11T23:03:32.4242302Z OK
2023-01-11T23:03:32.4242321Z 
2023-01-11T23:03:32.4242444Z Generating XML reports...
2023-01-11T23:03:32.4243001Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225751.xml
2023-01-11T23:03:32.4243378Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4243543Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4243929Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4244128Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4244639Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0tfmjl4k
2023-01-11T23:03:32.4244927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0tfmjl4k/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4244948Z 
2023-01-11T23:03:32.4245056Z Running tests...
2023-01-11T23:03:32.4245328Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4245695Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4246018Z   test_devices_option_mismatch_reverse (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4246229Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102759
2023-01-11T23:03:32.4246451Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102760
2023-01-11T23:03:32.4246671Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102761
2023-01-11T23:03:32.4246890Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102762
2023-01-11T23:03:32.4247271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4247497Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4247887Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4248080Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4248534Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4248700Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4249082Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4249331Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4249701Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4249879Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4250252Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4250441Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4250805Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4250966Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4251350Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4251541Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4251804Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf3uud66y
2023-01-11T23:03:32.4252080Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf3uud66y/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4252338Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsf5ngayd
2023-01-11T23:03:32.4252611Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsf5ngayd/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4252870Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr5k8pl7g
2023-01-11T23:03:32.4253141Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr5k8pl7g/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4253380Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp89tyqdv1
2023-01-11T23:03:32.4253654Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp89tyqdv1/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4253889Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4254117Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4254348Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4254578Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4254728Z fi_getinfo: -61
2023-01-11T23:03:32.4254868Z fi_getinfo: -61
2023-01-11T23:03:32.4254987Z fi_getinfo: -61
2023-01-11T23:03:32.4255122Z fi_getinfo: -61
2023-01-11T23:03:32.4255223Z ok (4.694s)
2023-01-11T23:03:32.4255243Z 
2023-01-11T23:03:32.4255513Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4255624Z Ran 1 test in 4.695s
2023-01-11T23:03:32.4255644Z 
2023-01-11T23:03:32.4255739Z OK
2023-01-11T23:03:32.4255759Z 
2023-01-11T23:03:32.4255881Z Generating XML reports...
2023-01-11T23:03:32.4256421Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225759.xml
2023-01-11T23:03:32.4256803Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4256981Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4257362Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4257555Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4257877Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe995mucp
2023-01-11T23:03:32.4258158Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe995mucp/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4258219Z 
2023-01-11T23:03:32.4258331Z Running tests...
2023-01-11T23:03:32.4258599Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4258947Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4259275Z   test_owner_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4259498Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 102946
2023-01-11T23:03:32.4259717Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 102947
2023-01-11T23:03:32.4259936Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 102948
2023-01-11T23:03:32.4260158Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 102949
2023-01-11T23:03:32.4260536Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4260717Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4261101Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4261278Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4261651Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4261826Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4262206Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4262401Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4262771Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4262953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4263333Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4263506Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4263874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4264049Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4264434Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4264623Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4264889Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsllm2bhh
2023-01-11T23:03:32.4265167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsllm2bhh/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4265429Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt_h12for
2023-01-11T23:03:32.4265698Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt_h12for/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4265939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_jxgb7p7
2023-01-11T23:03:32.4266210Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_jxgb7p7/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4266468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpf52fh5tr
2023-01-11T23:03:32.4266735Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpf52fh5tr/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4267018Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4267250Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4267529Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4267761Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4267893Z fi_getinfo: -61
2023-01-11T23:03:32.4267995Z ok (9.719s)
2023-01-11T23:03:32.4268014Z 
2023-01-11T23:03:32.4268282Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4268394Z Ran 1 test in 9.719s
2023-01-11T23:03:32.4268413Z 
2023-01-11T23:03:32.4268504Z OK
2023-01-11T23:03:32.4268523Z 
2023-01-11T23:03:32.4268646Z Generating XML reports...
2023-01-11T23:03:32.4269208Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225806.xml
2023-01-11T23:03:32.4269586Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4269769Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4270137Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4270332Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4270590Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpiggmmvcj
2023-01-11T23:03:32.4270865Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpiggmmvcj/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4270885Z 
2023-01-11T23:03:32.4270994Z Running tests...
2023-01-11T23:03:32.4271259Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4271625Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4271954Z   test_owner_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4272166Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103161
2023-01-11T23:03:32.4272390Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103162
2023-01-11T23:03:32.4272608Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103163
2023-01-11T23:03:32.4272827Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103164
2023-01-11T23:03:32.4273202Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4273381Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4273768Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4273961Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4274332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4274493Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4274871Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4275062Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4275430Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4275606Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4275982Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4276220Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4276600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4276803Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4277190Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4277379Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4277643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa2sy1u7t
2023-01-11T23:03:32.4277920Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa2sy1u7t/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4278179Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz1ikq1sr
2023-01-11T23:03:32.4278457Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz1ikq1sr/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4278716Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxuv2p9ky
2023-01-11T23:03:32.4278992Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxuv2p9ky/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4279230Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbxxruao_
2023-01-11T23:03:32.4279500Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbxxruao_/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4279731Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4279957Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4280187Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4280417Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4280569Z fi_getinfo: -61
2023-01-11T23:03:32.4280670Z ok (10.804s)
2023-01-11T23:03:32.4280689Z 
2023-01-11T23:03:32.4280941Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4281058Z Ran 1 test in 10.804s
2023-01-11T23:03:32.4281077Z 
2023-01-11T23:03:32.4281170Z OK
2023-01-11T23:03:32.4281188Z 
2023-01-11T23:03:32.4281311Z Generating XML reports...
2023-01-11T23:03:32.4281870Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225819.xml
2023-01-11T23:03:32.4282247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4282426Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4282811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4283008Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4283248Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpaf0kl0zl
2023-01-11T23:03:32.4283524Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpaf0kl0zl/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4283544Z 
2023-01-11T23:03:32.4283652Z Running tests...
2023-01-11T23:03:32.4283916Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4284512Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4284859Z   test_owner_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4285085Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103377
2023-01-11T23:03:32.4285306Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103378
2023-01-11T23:03:32.4285588Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103379
2023-01-11T23:03:32.4285814Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103380
2023-01-11T23:03:32.4286307Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4286486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4286870Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4287064Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4287435Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4287612Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4288001Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4288176Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4288544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4288726Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4289102Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4289291Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4289653Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4289828Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4290212Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4290387Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4290650Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpi681e0a1
2023-01-11T23:03:32.4290927Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpi681e0a1/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4291185Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzcr2lz7m
2023-01-11T23:03:32.4291459Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzcr2lz7m/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4291715Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3ok15y22
2023-01-11T23:03:32.4291982Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3ok15y22/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4292236Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpurc2dzgi
2023-01-11T23:03:32.4292508Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpurc2dzgi/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4292723Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4292954Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4293182Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4293413Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4293560Z fi_getinfo: -61
2023-01-11T23:03:32.4293660Z ok (10.989s)
2023-01-11T23:03:32.4293680Z 
2023-01-11T23:03:32.4293946Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4294058Z Ran 1 test in 10.989s
2023-01-11T23:03:32.4294078Z 
2023-01-11T23:03:32.4294152Z OK
2023-01-11T23:03:32.4294170Z 
2023-01-11T23:03:32.4294294Z Generating XML reports...
2023-01-11T23:03:32.4294899Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225832.xml
2023-01-11T23:03:32.4295287Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4295511Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4295899Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4296093Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4296350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk3lzo_ig
2023-01-11T23:03:32.4296621Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk3lzo_ig/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4296642Z 
2023-01-11T23:03:32.4296731Z Running tests...
2023-01-11T23:03:32.4297002Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4297367Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4297699Z   test_owner_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4297923Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103593
2023-01-11T23:03:32.4298145Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103594
2023-01-11T23:03:32.4298364Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103595
2023-01-11T23:03:32.4298581Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103596
2023-01-11T23:03:32.4298941Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4299119Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4299506Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4299701Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4300078Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4300254Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4300632Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4300825Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4301194Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4301354Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4301734Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4301926Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4302296Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4302469Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4302853Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4303043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4303306Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr11cfx7y
2023-01-11T23:03:32.4303564Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr11cfx7y/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4303885Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzdkhv3eh
2023-01-11T23:03:32.4304167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzdkhv3eh/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4304473Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp30k7hizr
2023-01-11T23:03:32.4304742Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp30k7hizr/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4304999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp26rds4fp
2023-01-11T23:03:32.4305268Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp26rds4fp/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4305500Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4305724Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4305937Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4306171Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4306323Z fi_getinfo: -61
2023-01-11T23:03:32.4306427Z ok (9.684s)
2023-01-11T23:03:32.4306447Z 
2023-01-11T23:03:32.4306714Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4306825Z Ran 1 test in 9.684s
2023-01-11T23:03:32.4306845Z 
2023-01-11T23:03:32.4306935Z OK
2023-01-11T23:03:32.4306954Z 
2023-01-11T23:03:32.4307080Z Generating XML reports...
2023-01-11T23:03:32.4307617Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225846.xml
2023-01-11T23:03:32.4307992Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4308169Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4308558Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4308754Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4309017Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmgy9y8xt
2023-01-11T23:03:32.4309288Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmgy9y8xt/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4309308Z 
2023-01-11T23:03:32.4309418Z Running tests...
2023-01-11T23:03:32.4309681Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4310031Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4310348Z   test_rref_as_arg_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4310570Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 103808
2023-01-11T23:03:32.4310797Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 103809
2023-01-11T23:03:32.4311018Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 103810
2023-01-11T23:03:32.4311239Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 103811
2023-01-11T23:03:32.4311614Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4311794Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4312163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4312357Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4312729Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4312953Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4313344Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4313580Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4313954Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4314130Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4314504Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4314677Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4315042Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4315216Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4315599Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4315791Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4316056Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpxkr47n3k
2023-01-11T23:03:32.4316330Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpxkr47n3k/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4316588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppkd1j3x6
2023-01-11T23:03:32.4316843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppkd1j3x6/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4317102Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppy7cg4av
2023-01-11T23:03:32.4317372Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppy7cg4av/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4317630Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpcorjy_j4
2023-01-11T23:03:32.4317900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpcorjy_j4/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4318135Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4318357Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4318586Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4318812Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4318942Z fi_getinfo: -61
2023-01-11T23:03:32.4319080Z fi_getinfo: -61
2023-01-11T23:03:32.4319215Z fi_getinfo: -61
2023-01-11T23:03:32.4319350Z fi_getinfo: -61
2023-01-11T23:03:32.4319449Z ok (15.898s)
2023-01-11T23:03:32.4319469Z 
2023-01-11T23:03:32.4319736Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4319853Z Ran 1 test in 15.898s
2023-01-11T23:03:32.4319873Z 
2023-01-11T23:03:32.4319948Z OK
2023-01-11T23:03:32.4319966Z 
2023-01-11T23:03:32.4320094Z Generating XML reports...
2023-01-11T23:03:32.4320657Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225858.xml
2023-01-11T23:03:32.4321035Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4321214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4321600Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4321795Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4322053Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprljriki2
2023-01-11T23:03:32.4322376Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprljriki2/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4322397Z 
2023-01-11T23:03:32.4322492Z Running tests...
2023-01-11T23:03:32.4322809Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4323175Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4323543Z   test_rref_as_arg_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4323768Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104155
2023-01-11T23:03:32.4323991Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104156
2023-01-11T23:03:32.4324417Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104157
2023-01-11T23:03:32.4324653Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104158
2023-01-11T23:03:32.4325027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4325205Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4325595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4325789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4326163Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4326343Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4326722Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4326911Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4327280Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4327440Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4327822Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4328010Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4328375Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4328553Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4328934Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4329123Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4329387Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp958yjq9
2023-01-11T23:03:32.4329641Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp958yjq9/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4329899Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe3bcpoku
2023-01-11T23:03:32.4330177Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe3bcpoku/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4330435Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpj7hamebn
2023-01-11T23:03:32.4330709Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpj7hamebn/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4330966Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp2v96tyc
2023-01-11T23:03:32.4331238Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp2v96tyc/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4331470Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4331772Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4331995Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4332280Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4332434Z fi_getinfo: -61
2023-01-11T23:03:32.4332573Z fi_getinfo: -61
2023-01-11T23:03:32.4332708Z fi_getinfo: -61
2023-01-11T23:03:32.4332844Z fi_getinfo: -61
2023-01-11T23:03:32.4332945Z ok (17.514s)
2023-01-11T23:03:32.4332965Z 
2023-01-11T23:03:32.4333212Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4333326Z Ran 1 test in 17.514s
2023-01-11T23:03:32.4333345Z 
2023-01-11T23:03:32.4333437Z OK
2023-01-11T23:03:32.4333456Z 
2023-01-11T23:03:32.4333579Z Generating XML reports...
2023-01-11T23:03:32.4334143Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225917.xml
2023-01-11T23:03:32.4334521Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4334704Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4335091Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4335284Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4335525Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptb5lwizu
2023-01-11T23:03:32.4335800Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptb5lwizu/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4335820Z 
2023-01-11T23:03:32.4335930Z Running tests...
2023-01-11T23:03:32.4336193Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4336564Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4336885Z   test_rref_as_arg_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4337666Z skip: Test is disabled because an issue exists disabling it: https://github.com/pytorch/pytorch/issues/81962 for platform(s) linux. If you're seeing this on your local machine and would like to enable this test, please make sure CI is not set and you are not using the flag --import-disabled-tests. (1.563s)
2023-01-11T23:03:32.4337687Z 
2023-01-11T23:03:32.4337952Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4338064Z Ran 1 test in 1.563s
2023-01-11T23:03:32.4338083Z 
2023-01-11T23:03:32.4338173Z OK (skipped=1)
2023-01-11T23:03:32.4338210Z 
2023-01-11T23:03:32.4338316Z Generating XML reports...
2023-01-11T23:03:32.4338875Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225937.xml
2023-01-11T23:03:32.4339255Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4339437Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4339824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4340019Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4340279Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp77p_vbmn
2023-01-11T23:03:32.4340552Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp77p_vbmn/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4340572Z 
2023-01-11T23:03:32.4340663Z Running tests...
2023-01-11T23:03:32.4340928Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4341340Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4341666Z   test_rref_as_arg_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4341935Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104542
2023-01-11T23:03:32.4342158Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104543
2023-01-11T23:03:32.4342377Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104544
2023-01-11T23:03:32.4342596Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104545
2023-01-11T23:03:32.4342979Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4343139Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4343529Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4343722Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4344099Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4344276Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4344655Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4344848Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4345216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4345376Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4345757Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4345946Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4346312Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4346492Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4346875Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4347067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4347328Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmphaqyten0
2023-01-11T23:03:32.4347587Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp90tau6fi
2023-01-11T23:03:32.4347847Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmphaqyten0/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4348114Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp90tau6fi/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4348373Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptvncllhj
2023-01-11T23:03:32.4348648Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptvncllhj/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4348903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsh57228j
2023-01-11T23:03:32.4349170Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsh57228j/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4349404Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4349631Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4349844Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4350120Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4350281Z fi_getinfo: -61
2023-01-11T23:03:32.4350420Z fi_getinfo: -61
2023-01-11T23:03:32.4350556Z fi_getinfo: -61
2023-01-11T23:03:32.4350743Z fi_getinfo: -61
2023-01-11T23:03:32.4350842Z ok (17.378s)
2023-01-11T23:03:32.4350862Z 
2023-01-11T23:03:32.4351126Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4351223Z Ran 1 test in 17.378s
2023-01-11T23:03:32.4351242Z 
2023-01-11T23:03:32.4351335Z OK
2023-01-11T23:03:32.4351354Z 
2023-01-11T23:03:32.4351479Z Generating XML reports...
2023-01-11T23:03:32.4352038Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225942.xml
2023-01-11T23:03:32.4352414Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4352590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4352974Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4353168Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4353414Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpitabhz0f
2023-01-11T23:03:32.4353687Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpitabhz0f/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4353707Z 
2023-01-11T23:03:32.4353817Z Running tests...
2023-01-11T23:03:32.4354083Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4354448Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4354767Z   test_rref_as_arg_synchronization5 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4354991Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 104895
2023-01-11T23:03:32.4355214Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 104896
2023-01-11T23:03:32.4355433Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 104897
2023-01-11T23:03:32.4355638Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 104898
2023-01-11T23:03:32.4356019Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4356198Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4356584Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4356778Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4357149Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4357329Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4357707Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4357882Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4358250Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4358427Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4358801Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4358989Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4359353Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4359578Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4359970Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4360203Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4360444Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0_vo1eb8
2023-01-11T23:03:32.4360716Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0_vo1eb8/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4360973Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpzax71ypa
2023-01-11T23:03:32.4361247Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpzax71ypa/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4361506Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp47xokqid
2023-01-11T23:03:32.4361784Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp47xokqid/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4362043Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0yw2pgbg
2023-01-11T23:03:32.4362314Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0yw2pgbg/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4362532Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4362756Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4362987Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4363216Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4363367Z fi_getinfo: -61
2023-01-11T23:03:32.4363505Z fi_getinfo: -61
2023-01-11T23:03:32.4363640Z fi_getinfo: -61
2023-01-11T23:03:32.4363776Z fi_getinfo: -61
2023-01-11T23:03:32.4363858Z ok (16.203s)
2023-01-11T23:03:32.4363877Z 
2023-01-11T23:03:32.4364144Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4364486Z Ran 1 test in 16.203s
2023-01-11T23:03:32.4364508Z 
2023-01-11T23:03:32.4364605Z OK
2023-01-11T23:03:32.4364629Z 
2023-01-11T23:03:32.4364757Z Generating XML reports...
2023-01-11T23:03:32.4365327Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230002.xml
2023-01-11T23:03:32.4365706Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4365885Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4366249Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4366445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4366707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmptx5xsc0q
2023-01-11T23:03:32.4366984Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmptx5xsc0q/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4367007Z 
2023-01-11T23:03:32.4367115Z Running tests...
2023-01-11T23:03:32.4367379Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4367746Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4368069Z   test_rref_forward_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4368295Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105242
2023-01-11T23:03:32.4368497Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105243
2023-01-11T23:03:32.4368717Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 105244
2023-01-11T23:03:32.4369023Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 105245
2023-01-11T23:03:32.4369418Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4369658Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4370046Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4370241Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4370615Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4370773Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4371153Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4371343Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4371714Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4371891Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4372271Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4372461Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4372830Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4373005Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4373370Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4373560Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4373824Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbe_dn_3p
2023-01-11T23:03:32.4374102Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbe_dn_3p/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4374370Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpnf9tp0s6
2023-01-11T23:03:32.4374644Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpnf9tp0s6/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4374903Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdy659248
2023-01-11T23:03:32.4375169Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdy659248/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4375425Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpg62tctz3
2023-01-11T23:03:32.4375676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpg62tctz3/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4375909Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4376136Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4376368Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4376598Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4376745Z fi_getinfo: -61
2023-01-11T23:03:32.4376882Z fi_getinfo: -61
2023-01-11T23:03:32.4377001Z fi_getinfo: -61
2023-01-11T23:03:32.4377138Z fi_getinfo: -61
2023-01-11T23:03:32.4377240Z ok (15.192s)
2023-01-11T23:03:32.4377259Z 
2023-01-11T23:03:32.4377527Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4377638Z Ran 1 test in 15.192s
2023-01-11T23:03:32.4377658Z 
2023-01-11T23:03:32.4377749Z OK
2023-01-11T23:03:32.4377768Z 
2023-01-11T23:03:32.4377891Z Generating XML reports...
2023-01-11T23:03:32.4378502Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230021.xml
2023-01-11T23:03:32.4378874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4379100Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4379488Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4379683Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4379939Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_36a0e0_
2023-01-11T23:03:32.4380207Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_36a0e0_/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4380227Z 
2023-01-11T23:03:32.4380336Z Running tests...
2023-01-11T23:03:32.4380601Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4380968Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4381274Z   test_rref_forward_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4381504Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105588
2023-01-11T23:03:32.4381727Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105589
2023-01-11T23:03:32.4381946Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 105590
2023-01-11T23:03:32.4382164Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 105591
2023-01-11T23:03:32.4382544Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4382722Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4383111Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4383286Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4383664Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4383842Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4384222Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4384415Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4384785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4384963Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4385342Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4385533Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4385880Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4386055Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4386438Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4386626Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4386887Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8ykh3zj5
2023-01-11T23:03:32.4387163Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8ykh3zj5/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4387471Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8cegou1x
2023-01-11T23:03:32.4387750Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8cegou1x/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4388008Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp_dn95yi3
2023-01-11T23:03:32.4388302Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp_dn95yi3/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4388558Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp7zzp4uys
2023-01-11T23:03:32.4388831Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp7zzp4uys/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4389066Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4389292Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4389523Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4389757Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4389907Z fi_getinfo: -61
2023-01-11T23:03:32.4390029Z fi_getinfo: -61
2023-01-11T23:03:32.4390169Z fi_getinfo: -61
2023-01-11T23:03:32.4390304Z fi_getinfo: -61
2023-01-11T23:03:32.4390405Z ok (15.589s)
2023-01-11T23:03:32.4390425Z 
2023-01-11T23:03:32.4390691Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4390804Z Ran 1 test in 15.589s
2023-01-11T23:03:32.4390823Z 
2023-01-11T23:03:32.4390916Z OK
2023-01-11T23:03:32.4390935Z 
2023-01-11T23:03:32.4391040Z Generating XML reports...
2023-01-11T23:03:32.4391596Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230039.xml
2023-01-11T23:03:32.4391973Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4392155Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4392541Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4392739Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4392999Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpk22osj3u
2023-01-11T23:03:32.4393272Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpk22osj3u/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4393291Z 
2023-01-11T23:03:32.4393399Z Running tests...
2023-01-11T23:03:32.4393644Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4394012Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4394333Z   test_rref_forward_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4394562Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 105937
2023-01-11T23:03:32.4394786Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 105938
2023-01-11T23:03:32.4395009Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 105939
2023-01-11T23:03:32.4395230Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 105940
2023-01-11T23:03:32.4395608Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4395768Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4396154Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4396347Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4396770Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4396952Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4397335Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4397576Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4397949Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4398125Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4398482Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4398673Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4399037Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4399214Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4399595Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4399789Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4400051Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp8dolpfkb
2023-01-11T23:03:32.4400329Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp8dolpfkb/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4400588Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppjvo_fln
2023-01-11T23:03:32.4400843Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppjvo_fln/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4401100Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp6t6fi0rb
2023-01-11T23:03:32.4401373Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp6t6fi0rb/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4401627Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0a6vqsoa
2023-01-11T23:03:32.4401900Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0a6vqsoa/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4402134Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4402359Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4402590Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4402801Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4402953Z fi_getinfo: -61
2023-01-11T23:03:32.4403091Z fi_getinfo: -61
2023-01-11T23:03:32.4403225Z fi_getinfo: -61
2023-01-11T23:03:32.4403360Z fi_getinfo: -61
2023-01-11T23:03:32.4403462Z ok (15.613s)
2023-01-11T23:03:32.4403485Z 
2023-01-11T23:03:32.4403751Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4403847Z Ran 1 test in 15.614s
2023-01-11T23:03:32.4403887Z 
2023-01-11T23:03:32.4403963Z OK
2023-01-11T23:03:32.4403982Z 
2023-01-11T23:03:32.4404104Z Generating XML reports...
2023-01-11T23:03:32.4404928Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230057.xml
2023-01-11T23:03:32.4405310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4405491Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4405878Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4406073Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4406409Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0ug3x760
2023-01-11T23:03:32.4406670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0ug3x760/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4406759Z 
2023-01-11T23:03:32.4406854Z Running tests...
2023-01-11T23:03:32.4407122Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4407487Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4407808Z   test_rref_forward_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4408032Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106286
2023-01-11T23:03:32.4408258Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106287
2023-01-11T23:03:32.4408477Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106288
2023-01-11T23:03:32.4408681Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106289
2023-01-11T23:03:32.4409067Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4409249Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4409636Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4409830Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4410204Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4410380Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4410758Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4410954Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4411302Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4411484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4411860Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4412049Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4412413Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4412590Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4412968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4413159Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4413418Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp9qredfho
2023-01-11T23:03:32.4413676Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp9qredfho/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4413935Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprjnokekg
2023-01-11T23:03:32.4414208Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprjnokekg/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4414468Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmporhzjqn4
2023-01-11T23:03:32.4414740Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmporhzjqn4/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4414996Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpamufa211
2023-01-11T23:03:32.4415264Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpamufa211/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4415547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4415764Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4416038Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4416268Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4416420Z fi_getinfo: -61
2023-01-11T23:03:32.4416558Z fi_getinfo: -61
2023-01-11T23:03:32.4416695Z fi_getinfo: -61
2023-01-11T23:03:32.4416830Z fi_getinfo: -61
2023-01-11T23:03:32.4416913Z ok (15.593s)
2023-01-11T23:03:32.4416954Z 
2023-01-11T23:03:32.4417203Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4417316Z Ran 1 test in 15.593s
2023-01-11T23:03:32.4417335Z 
2023-01-11T23:03:32.4417425Z OK
2023-01-11T23:03:32.4417445Z 
2023-01-11T23:03:32.4417568Z Generating XML reports...
2023-01-11T23:03:32.4418131Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230115.xml
2023-01-11T23:03:32.4418513Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4418692Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4419076Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4419253Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4419515Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5c2fskwn
2023-01-11T23:03:32.4419790Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5c2fskwn/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4419810Z 
2023-01-11T23:03:32.4419918Z Running tests...
2023-01-11T23:03:32.4420188Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4420555Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4420880Z   test_rref_to_here_synchronization1 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4421106Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106632
2023-01-11T23:03:32.4421328Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106633
2023-01-11T23:03:32.4421531Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106634
2023-01-11T23:03:32.4421747Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106635
2023-01-11T23:03:32.4422123Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4422309Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4422698Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4422894Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4423266Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4423484Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4423851Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4424043Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4424410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4424588Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4425015Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4425216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4425624Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4425799Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4426181Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4426352Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4426615Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpyc3zrmtp
2023-01-11T23:03:32.4426892Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpyc3zrmtp/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4427157Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpt5axc0el
2023-01-11T23:03:32.4427431Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpt5axc0el/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4427694Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppm4gwr8x
2023-01-11T23:03:32.4427966Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppm4gwr8x/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4428223Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc0ycra61
2023-01-11T23:03:32.4428474Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc0ycra61/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4428705Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4428931Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4429166Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4429397Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4429544Z fi_getinfo: -61
2023-01-11T23:03:32.4429687Z fi_getinfo: -61
2023-01-11T23:03:32.4429825Z fi_getinfo: -61
2023-01-11T23:03:32.4429943Z fi_getinfo: -61
2023-01-11T23:03:32.4430043Z ok (15.908s)
2023-01-11T23:03:32.4430062Z 
2023-01-11T23:03:32.4430330Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4430444Z Ran 1 test in 15.908s
2023-01-11T23:03:32.4430463Z 
2023-01-11T23:03:32.4430555Z OK
2023-01-11T23:03:32.4430575Z 
2023-01-11T23:03:32.4430698Z Generating XML reports...
2023-01-11T23:03:32.4431257Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230134.xml
2023-01-11T23:03:32.4431639Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4431801Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4432185Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4432383Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4432643Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeg46_b9v
2023-01-11T23:03:32.4432919Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeg46_b9v/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4432939Z 
2023-01-11T23:03:32.4433047Z Running tests...
2023-01-11T23:03:32.4433313Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4433678Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4434097Z   test_rref_to_here_synchronization2 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4434311Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 106979
2023-01-11T23:03:32.4434535Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 106980
2023-01-11T23:03:32.4434800Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 106981
2023-01-11T23:03:32.4435019Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 106982
2023-01-11T23:03:32.4435403Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4435583Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4435968Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4436161Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4436517Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4436695Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4437075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4437266Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4437633Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4437813Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4438187Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4438363Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4438750Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4438925Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4439310Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4439503Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4439766Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp1op77uvx
2023-01-11T23:03:32.4440040Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp1op77uvx/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4440299Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp__8pgims
2023-01-11T23:03:32.4440569Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp__8pgims/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4440832Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpp12tfe_q
2023-01-11T23:03:32.4441094Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpp12tfe_q/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4441356Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpz789p8gh
2023-01-11T23:03:32.4441627Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpz789p8gh/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4441860Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4442087Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4442318Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4442549Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4442695Z fi_getinfo: -61
2023-01-11T23:03:32.4442816Z fi_getinfo: -61
2023-01-11T23:03:32.4442953Z fi_getinfo: -61
2023-01-11T23:03:32.4443087Z fi_getinfo: -61
2023-01-11T23:03:32.4443233Z ok (17.514s)
2023-01-11T23:03:32.4443255Z 
2023-01-11T23:03:32.4443528Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4443687Z Ran 1 test in 17.514s
2023-01-11T23:03:32.4443707Z 
2023-01-11T23:03:32.4443799Z OK
2023-01-11T23:03:32.4443819Z 
2023-01-11T23:03:32.4443943Z Generating XML reports...
2023-01-11T23:03:32.4444715Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230152.xml
2023-01-11T23:03:32.4445109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4445288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4445671Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4445870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4446132Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdk8p9851
2023-01-11T23:03:32.4446404Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdk8p9851/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4446427Z 
2023-01-11T23:03:32.4446535Z Running tests...
2023-01-11T23:03:32.4446800Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4447148Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4447470Z   test_rref_to_here_synchronization3 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4447692Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107332
2023-01-11T23:03:32.4447914Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107333
2023-01-11T23:03:32.4448137Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 107334
2023-01-11T23:03:32.4448357Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 107335
2023-01-11T23:03:32.4448739Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4448917Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4449288Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4449483Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4449854Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4450029Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4450410Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4450603Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4450972Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4451154Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4451527Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4451699Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4452063Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4452237Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4452619Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4452883Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4453159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmeddiz8g
2023-01-11T23:03:32.4453493Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmeddiz8g/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4453750Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwlmzc00y
2023-01-11T23:03:32.4453986Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpeldwygom
2023-01-11T23:03:32.4454258Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwlmzc00y/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4454527Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpeldwygom/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4454787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpc37utmix
2023-01-11T23:03:32.4455059Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpc37utmix/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4455291Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4455521Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4455754Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4455984Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4456118Z fi_getinfo: -61
2023-01-11T23:03:32.4456258Z fi_getinfo: -61
2023-01-11T23:03:32.4456394Z fi_getinfo: -61
2023-01-11T23:03:32.4456531Z fi_getinfo: -61
2023-01-11T23:03:32.4456632Z ok (16.109s)
2023-01-11T23:03:32.4456651Z 
2023-01-11T23:03:32.4456916Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4457029Z Ran 1 test in 16.109s
2023-01-11T23:03:32.4457049Z 
2023-01-11T23:03:32.4457125Z OK
2023-01-11T23:03:32.4457144Z 
2023-01-11T23:03:32.4457274Z Generating XML reports...
2023-01-11T23:03:32.4457834Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230213.xml
2023-01-11T23:03:32.4458216Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4458395Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4458778Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4458972Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4459231Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpsdou5tiy
2023-01-11T23:03:32.4459504Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpsdou5tiy/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4459524Z 
2023-01-11T23:03:32.4459619Z Running tests...
2023-01-11T23:03:32.4459883Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4460246Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4460571Z   test_rref_to_here_synchronization4 (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4460793Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 107679
2023-01-11T23:03:32.4461016Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 107680
2023-01-11T23:03:32.4461236Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 107681
2023-01-11T23:03:32.4461454Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 107682
2023-01-11T23:03:32.4461814Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4462044Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4462443Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4462684Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4463059Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4463236Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4463616Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4463807Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4464175Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4464339Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4464716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4464907Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4465269Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4465444Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4465824Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4466013Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4466275Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprogkwkd9
2023-01-11T23:03:32.4466533Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprogkwkd9/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4466796Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp3u1kxnx0
2023-01-11T23:03:32.4467068Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp3u1kxnx0/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4467332Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpr2oibpqb
2023-01-11T23:03:32.4467606Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpr2oibpqb/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4467865Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpawbe38ks
2023-01-11T23:03:32.4468136Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpawbe38ks/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4468369Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4468594Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4468810Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4469043Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4469196Z fi_getinfo: -61
2023-01-11T23:03:32.4469334Z fi_getinfo: -61
2023-01-11T23:03:32.4469472Z fi_getinfo: -61
2023-01-11T23:03:32.4469607Z fi_getinfo: -61
2023-01-11T23:03:32.4469706Z ok (17.070s)
2023-01-11T23:03:32.4469728Z 
2023-01-11T23:03:32.4469977Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4470091Z Ran 1 test in 17.071s
2023-01-11T23:03:32.4470110Z 
2023-01-11T23:03:32.4470201Z OK
2023-01-11T23:03:32.4470221Z 
2023-01-11T23:03:32.4470344Z Generating XML reports...
2023-01-11T23:03:32.4470905Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230231.xml
2023-01-11T23:03:32.4471347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4471535Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4471925Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4472169Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4472413Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmiin_ntp
2023-01-11T23:03:32.4472689Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmiin_ntp/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4472709Z 
2023-01-11T23:03:32.4472816Z Running tests...
2023-01-11T23:03:32.4473082Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4473450Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4473781Z   test_rref_with_unpickleable_attributes (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4474006Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108032
2023-01-11T23:03:32.4474230Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108033
2023-01-11T23:03:32.4474431Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 108034
2023-01-11T23:03:32.4474650Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 108035
2023-01-11T23:03:32.4475027Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4475204Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4475591Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4475785Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4476162Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4476337Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4476716Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4476891Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4477258Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4477436Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4477811Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4478003Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4478368Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4478544Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4478930Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4479101Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4479363Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmplghr85vu
2023-01-11T23:03:32.4479637Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmplghr85vu/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4479894Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp5u5a18z2
2023-01-11T23:03:32.4480167Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp5u5a18z2/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4480475Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp67qeodbx
2023-01-11T23:03:32.4480754Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp67qeodbx/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4481055Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpn0vrtf2k
2023-01-11T23:03:32.4481326Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpn0vrtf2k/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4481538Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4481766Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4481996Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4482227Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4482376Z fi_getinfo: -61
2023-01-11T23:03:32.4482514Z fi_getinfo: -61
2023-01-11T23:03:32.4482658Z fi_getinfo: -61
2023-01-11T23:03:32.4482776Z fi_getinfo: -61
2023-01-11T23:03:32.4482878Z ok (6.616s)
2023-01-11T23:03:32.4482898Z 
2023-01-11T23:03:32.4483162Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4483277Z Ran 1 test in 6.616s
2023-01-11T23:03:32.4483296Z 
2023-01-11T23:03:32.4483388Z OK
2023-01-11T23:03:32.4483408Z 
2023-01-11T23:03:32.4483532Z Generating XML reports...
2023-01-11T23:03:32.4484085Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230251.xml
2023-01-11T23:03:32.4484692Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4484876Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4485247Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4485445Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4485707Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbndj0km3
2023-01-11T23:03:32.4485989Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbndj0km3/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4486009Z 
2023-01-11T23:03:32.4486117Z Running tests...
2023-01-11T23:03:32.4486380Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4486746Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4487060Z   test_tensor_view_as_return_value (__main__.TensorPipeTensorPipeAgentCudaRpcTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4487267Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 108383
2023-01-11T23:03:32.4487487Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 108384
2023-01-11T23:03:32.4487710Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 108385
2023-01-11T23:03:32.4487927Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 108386
2023-01-11T23:03:32.4488308Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4488486Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4488874Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4489067Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4489437Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4489597Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4490052Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4490252Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4490679Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4490856Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4491229Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4491417Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4491783Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4491939Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4492327Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4492516Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4492779Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprxrqv8nl
2023-01-11T23:03:32.4493062Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprxrqv8nl/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4493318Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpo0z_lcod
2023-01-11T23:03:32.4493589Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpo0z_lcod/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4493844Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpa_tnq3tr
2023-01-11T23:03:32.4494113Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpa_tnq3tr/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4494350Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpjlcr_mny
2023-01-11T23:03:32.4494622Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpjlcr_mny/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4494856Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4495085Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4495317Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4495547Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4495694Z fi_getinfo: -61
2023-01-11T23:03:32.4495832Z fi_getinfo: -61
2023-01-11T23:03:32.4495950Z fi_getinfo: -61
2023-01-11T23:03:32.4496087Z fi_getinfo: -61
2023-01-11T23:03:32.4496187Z ok (8.913s)
2023-01-11T23:03:32.4496207Z 
2023-01-11T23:03:32.4496474Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4496585Z Ran 1 test in 8.913s
2023-01-11T23:03:32.4496605Z 
2023-01-11T23:03:32.4496700Z OK
2023-01-11T23:03:32.4496719Z 
2023-01-11T23:03:32.4496842Z Generating XML reports...
2023-01-11T23:03:32.4497406Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230301.xml
2023-01-11T23:03:32.4497772Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4497951Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4498332Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4498528Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4498787Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpbpquo2zy
2023-01-11T23:03:32.4499125Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpbpquo2zy/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4499146Z 
2023-01-11T23:03:32.4499263Z Running tests...
2023-01-11T23:03:32.4499529Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4499925Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4500255Z   test_device_maps_backward_pass (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4500479Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109030
2023-01-11T23:03:32.4500701Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109031
2023-01-11T23:03:32.4500923Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 109032
2023-01-11T23:03:32.4501141Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 109033
2023-01-11T23:03:32.4501525Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4501702Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4502088Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4502267Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4502640Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4502815Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4503192Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4503385Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4503752Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4503933Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4504309Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4504482Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4504846Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4505018Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4505400Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4505588Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4505851Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwvlnyc5_
2023-01-11T23:03:32.4506131Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwvlnyc5_/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4506390Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpdoluvtwc
2023-01-11T23:03:32.4506670Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpdoluvtwc/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4506911Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpv37l0kx9
2023-01-11T23:03:32.4507180Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpv37l0kx9/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4507434Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0xyyyjzy
2023-01-11T23:03:32.4507706Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0xyyyjzy/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4507939Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4508220Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4508459Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4508686Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4508863Z skip: Need at least 4 CUDA devices (4.353s)
2023-01-11T23:03:32.4508903Z 
2023-01-11T23:03:32.4509156Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4509268Z Ran 1 test in 4.354s
2023-01-11T23:03:32.4509287Z 
2023-01-11T23:03:32.4509394Z OK (skipped=1)
2023-01-11T23:03:32.4509413Z 
2023-01-11T23:03:32.4509536Z Generating XML reports...
2023-01-11T23:03:32.4510113Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20230111230312.xml
2023-01-11T23:03:32.4510493Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4510677Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4511065Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4511245Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4511504Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp143wzs3u
2023-01-11T23:03:32.4511775Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp143wzs3u/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4511794Z 
2023-01-11T23:03:32.4511903Z Running tests...
2023-01-11T23:03:32.4512164Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4512530Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4512861Z   test_dist_autograd_sync_streams (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4513087Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109201
2023-01-11T23:03:32.4513307Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109202
2023-01-11T23:03:32.4513511Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 109203
2023-01-11T23:03:32.4513730Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 109204
2023-01-11T23:03:32.4514109Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4514288Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4514674Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4514870Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4515248Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4515424Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4515785Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4515977Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4516347Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4516524Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4516898Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4517087Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4517501Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4517682Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4518070Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4518285Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4518548Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpe79ru6ep
2023-01-11T23:03:32.4518822Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpe79ru6ep/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4519081Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp26o87zwg
2023-01-11T23:03:32.4519354Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp26o87zwg/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4519613Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmprz8u8xj9
2023-01-11T23:03:32.4519885Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmprz8u8xj9/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4520139Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpq7l7qbvu
2023-01-11T23:03:32.4520411Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpq7l7qbvu/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4520626Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4520850Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4521081Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4521309Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4521462Z skip: Need at least 4 CUDA devices (4.466s)
2023-01-11T23:03:32.4521482Z 
2023-01-11T23:03:32.4521755Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4521871Z Ran 1 test in 4.467s
2023-01-11T23:03:32.4521891Z 
2023-01-11T23:03:32.4521998Z OK (skipped=1)
2023-01-11T23:03:32.4522018Z 
2023-01-11T23:03:32.4522127Z Generating XML reports...
2023-01-11T23:03:32.4522712Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20230111230319.xml
2023-01-11T23:03:32.4523087Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4523265Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4523699Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4523898Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4524159Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpmrneax7n
2023-01-11T23:03:32.4524645Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpmrneax7n/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4524667Z 
2023-01-11T23:03:32.4524780Z Running tests...
2023-01-11T23:03:32.4525038Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4525404Z Test results will be stored in test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent
2023-01-11T23:03:32.4525736Z   test_gradients_synchronizations (__main__.TensorPipeTensorPipeCudaDistAutogradTest) ... INFO:numba.cuda.cudadrv.driver:init
2023-01-11T23:03:32.4525957Z INFO:torch.testing._internal.common_distributed:Started process 0 with pid 109372
2023-01-11T23:03:32.4526180Z INFO:torch.testing._internal.common_distributed:Started process 1 with pid 109373
2023-01-11T23:03:32.4526403Z INFO:torch.testing._internal.common_distributed:Started process 2 with pid 109374
2023-01-11T23:03:32.4526620Z INFO:torch.testing._internal.common_distributed:Started process 3 with pid 109375
2023-01-11T23:03:32.4527075Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4527244Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4527693Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4527888Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4528261Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4528439Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4528819Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4529007Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4529377Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4529555Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4529917Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4530107Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4530469Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:122: UserWarning: loaded 76 slow tests
2023-01-11T23:03:32.4530645Z   warnings.warn(f"loaded {len(slow_tests_dict)} slow tests")
2023-01-11T23:03:32.4531028Z /opt/conda/lib/python3.10/site-packages/torch/testing/_internal/common_utils.py:126: UserWarning: loaded 210 disabled tests
2023-01-11T23:03:32.4531216Z   warnings.warn(f"loaded {len(disabled_tests_dict)} disabled tests")
2023-01-11T23:03:32.4531480Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmp0s5h59az
2023-01-11T23:03:32.4531752Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmp0s5h59az/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4531995Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpwggammoq
2023-01-11T23:03:32.4532270Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpwggammoq/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4532529Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmpb6t7cfoc
2023-01-11T23:03:32.4532803Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmpb6t7cfoc/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4533061Z INFO:torch.distributed.nn.jit.instantiator:Created a temporary directory at /tmp/tmppnyiwmf1
2023-01-11T23:03:32.4533333Z INFO:torch.distributed.nn.jit.instantiator:Writing /tmp/tmppnyiwmf1/_remote_module_non_scriptable.py
2023-01-11T23:03:32.4533568Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 3
2023-01-11T23:03:32.4533792Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 2
2023-01-11T23:03:32.4534025Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 1
2023-01-11T23:03:32.4534242Z INFO:torch.testing._internal.common_distributed:Starting event listener thread for rank 0
2023-01-11T23:03:32.4534394Z skip: Need at least 4 CUDA devices (4.484s)
2023-01-11T23:03:32.4534414Z 
2023-01-11T23:03:32.4534686Z ----------------------------------------------------------------------
2023-01-11T23:03:32.4534799Z Ran 1 test in 4.485s
2023-01-11T23:03:32.4534819Z 
2023-01-11T23:03:32.4534927Z OK (skipped=1)
2023-01-11T23:03:32.4534946Z 
2023-01-11T23:03:32.4535069Z Generating XML reports...
2023-01-11T23:03:32.4535648Z Generated XML report: test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20230111230327.xml
2023-01-11T23:03:32.4535667Z 
2023-01-11T23:03:32.4536233Z ##[endgroup]
2023-01-11T23:03:32.4536761Z FINISHED PRINTING LOG FILE of distributed/rpc/cuda/test_tensorpipe_agent (/var/lib/jenkins/workspace/test/test-reports/distributed-rpc-cuda-test_tensorpipe_agent_mppcsvm1)
2023-01-11T23:03:32.4536846Z 
2023-01-11T23:03:32.6086392Z 
2023-01-11T23:03:32.6086961Z real	100m30.574s
2023-01-11T23:03:32.6087090Z user	168m34.986s
2023-01-11T23:03:32.6087189Z sys	93m44.797s
2023-01-11T23:03:32.6087307Z + assert_git_not_dirty
2023-01-11T23:03:32.6087697Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 != *rocm* ]]
2023-01-11T23:03:32.6087938Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 != *xla* ]]
2023-01-11T23:03:32.6091087Z ++ git status --porcelain
2023-01-11T23:03:33.6584567Z + git_status=
2023-01-11T23:03:33.6584828Z + [[ -n '' ]]
2023-01-11T23:03:33.6585112Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 == *cuda* ]]
2023-01-11T23:03:33.6585200Z + [[ 1 == 1 ]]
2023-01-11T23:03:33.6585390Z + echo 'Testing distributed C++ tests'
2023-01-11T23:03:33.6585530Z Testing distributed C++ tests
2023-01-11T23:03:33.6586779Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2023-01-11T23:03:33.6600932Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2023-01-11T23:03:33.6612102Z + TEST_REPORTS_DIR=test/test-reports/cpp-distributed/test_distributed
2023-01-11T23:03:33.6612384Z + mkdir -p test/test-reports/cpp-distributed/test_distributed
2023-01-11T23:03:33.6627298Z + /opt/conda/lib/python3.10/site-packages/torch/bin/FileStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml
2023-01-11T23:03:33.9979254Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2023-01-11T23:03:33.9979861Z [==========] Running 4 tests from 1 test suite.
2023-01-11T23:03:33.9980271Z [----------] Global test environment set-up.
2023-01-11T23:03:33.9980669Z [----------] 4 tests from FileStoreTest
2023-01-11T23:03:33.9981053Z [ RUN      ] FileStoreTest.testGetAndSet
2023-01-11T23:03:33.9984944Z [       OK ] FileStoreTest.testGetAndSet (0 ms)
2023-01-11T23:03:33.9985430Z [ RUN      ] FileStoreTest.testGetAndSetWithPrefix
2023-01-11T23:03:33.9990018Z [       OK ] FileStoreTest.testGetAndSetWithPrefix (0 ms)
2023-01-11T23:03:33.9990485Z [ RUN      ] FileStoreTest.testStressStore
2023-01-11T23:03:34.0200432Z [       OK ] FileStoreTest.testStressStore (20 ms)
2023-01-11T23:03:34.0200929Z [ RUN      ] FileStoreTest.testStressStoreWithPrefix
2023-01-11T23:03:34.0410061Z [       OK ] FileStoreTest.testStressStoreWithPrefix (20 ms)
2023-01-11T23:03:34.0410550Z [----------] 4 tests from FileStoreTest (43 ms total)
2023-01-11T23:03:34.0410774Z 
2023-01-11T23:03:34.0411010Z [----------] Global test environment tear-down
2023-01-11T23:03:34.0411802Z [==========] 4 tests from 1 test suite ran. (43 ms total)
2023-01-11T23:03:34.0412142Z [  PASSED  ] 4 tests.
2023-01-11T23:03:34.1090511Z + /opt/conda/lib/python3.10/site-packages/torch/bin/HashStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml
2023-01-11T23:03:34.4469761Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2023-01-11T23:03:34.4470759Z [==========] Running 4 tests from 1 test suite.
2023-01-11T23:03:34.4471208Z [----------] Global test environment set-up.
2023-01-11T23:03:34.4471610Z [----------] 4 tests from HashStoreTest
2023-01-11T23:03:34.4472113Z [ RUN      ] HashStoreTest.testGetAndSet
2023-01-11T23:03:34.5475720Z [       OK ] HashStoreTest.testGetAndSet (100 ms)
2023-01-11T23:03:34.5476231Z [ RUN      ] HashStoreTest.testGetAndSetWithPrefix
2023-01-11T23:03:34.6480403Z [       OK ] HashStoreTest.testGetAndSetWithPrefix (100 ms)
2023-01-11T23:03:34.6480890Z [ RUN      ] HashStoreTest.testStressStore
2023-01-11T23:03:34.6487112Z [       OK ] HashStoreTest.testStressStore (0 ms)
2023-01-11T23:03:34.6487608Z [ RUN      ] HashStoreTest.testStressStoreWithPrefix
2023-01-11T23:03:34.6494590Z [       OK ] HashStoreTest.testStressStoreWithPrefix (0 ms)
2023-01-11T23:03:34.6495096Z [----------] 4 tests from HashStoreTest (202 ms total)
2023-01-11T23:03:34.6495323Z 
2023-01-11T23:03:34.6495567Z [----------] Global test environment tear-down
2023-01-11T23:03:34.6498219Z [==========] 4 tests from 1 test suite ran. (202 ms total)
2023-01-11T23:03:34.6498601Z [  PASSED  ] 4 tests.
2023-01-11T23:03:34.7203161Z + /opt/conda/lib/python3.10/site-packages/torch/bin/TCPStoreTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml
2023-01-11T23:03:35.0561205Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2023-01-11T23:03:35.0561887Z [==========] Running 11 tests from 1 test suite.
2023-01-11T23:03:35.0562296Z [----------] Global test environment set-up.
2023-01-11T23:03:35.0562707Z [----------] 11 tests from TCPStoreTest
2023-01-11T23:03:35.0563099Z [ RUN      ] TCPStoreTest.testHelper
2023-01-11T23:03:36.0245605Z [       OK ] TCPStoreTest.testHelper (968 ms)
2023-01-11T23:03:36.0246107Z [ RUN      ] TCPStoreTest.testHelperPrefix
2023-01-11T23:03:37.0285803Z [       OK ] TCPStoreTest.testHelperPrefix (1003 ms)
2023-01-11T23:03:37.0286310Z [ RUN      ] TCPStoreTest.testWatchKeyCallback
2023-01-11T23:03:37.0424376Z [       OK ] TCPStoreTest.testWatchKeyCallback (13 ms)
2023-01-11T23:03:37.0424943Z [ RUN      ] TCPStoreTest.testWatchKeyCallbackWithPrefix
2023-01-11T23:03:37.0566081Z [       OK ] TCPStoreTest.testWatchKeyCallbackWithPrefix (14 ms)
2023-01-11T23:03:37.0566620Z [ RUN      ] TCPStoreTest.testKeyEmptyUpdate
2023-01-11T23:03:37.2653980Z [       OK ] TCPStoreTest.testKeyEmptyUpdate (208 ms)
2023-01-11T23:03:37.2654475Z [ RUN      ] TCPStoreTest.testKeyUpdate
2023-01-11T23:03:37.2662689Z [       OK ] TCPStoreTest.testKeyUpdate (0 ms)
2023-01-11T23:03:37.2663135Z [ RUN      ] TCPStoreTest.testKeyCreate
2023-01-11T23:03:37.2669828Z [       OK ] TCPStoreTest.testKeyCreate (0 ms)
2023-01-11T23:03:37.2670245Z [ RUN      ] TCPStoreTest.testKeyAdd
2023-01-11T23:03:37.2677061Z [       OK ] TCPStoreTest.testKeyAdd (0 ms)
2023-01-11T23:03:37.2677495Z [ RUN      ] TCPStoreTest.testKeyDelete
2023-01-11T23:03:37.4731427Z [       OK ] TCPStoreTest.testKeyDelete (205 ms)
2023-01-11T23:03:37.4731862Z [ RUN      ] TCPStoreTest.testCleanShutdown
2023-01-11T23:03:37.4739842Z [       OK ] TCPStoreTest.testCleanShutdown (0 ms)
2023-01-11T23:03:37.4740322Z [ RUN      ] TCPStoreTest.testMultiTenantStores
2023-01-11T23:03:37.4753383Z [       OK ] TCPStoreTest.testMultiTenantStores (1 ms)
2023-01-11T23:03:37.4753865Z [----------] 11 tests from TCPStoreTest (2419 ms total)
2023-01-11T23:03:37.4754095Z 
2023-01-11T23:03:37.4754329Z [----------] Global test environment tear-down
2023-01-11T23:03:37.4757127Z [==========] 11 tests from 1 test suite ran. (2419 ms total)
2023-01-11T23:03:37.4757508Z [  PASSED  ] 11 tests.
2023-01-11T23:03:37.5520952Z ++ command -v mpiexec
2023-01-11T23:03:37.5522858Z + MPIEXEC=/usr/bin/mpiexec
2023-01-11T23:03:37.5523567Z + [[ -n /usr/bin/mpiexec ]]
2023-01-11T23:03:37.5524744Z + [[ -z true ]]
2023-01-11T23:03:37.5525446Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupGlooTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml
2023-01-11T23:03:37.8914511Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2023-01-11T23:03:37.8915139Z [==========] Running 12 tests from 1 test suite.
2023-01-11T23:03:37.8915563Z [----------] Global test environment set-up.
2023-01-11T23:03:37.8915980Z [----------] 12 tests from ProcessGroupGlooTest
2023-01-11T23:03:37.8916465Z [ RUN      ] ProcessGroupGlooTest.testSIGSTOPException
2023-01-11T23:03:38.9584752Z [       OK ] ProcessGroupGlooTest.testSIGSTOPException (1066 ms)
2023-01-11T23:03:38.9585374Z [ RUN      ] ProcessGroupGlooTest.testSIGKILLException
2023-01-11T23:03:38.9916506Z [       OK ] ProcessGroupGlooTest.testSIGKILLException (33 ms)
2023-01-11T23:03:38.9917083Z [ RUN      ] ProcessGroupGlooTest.testAllReduceCPU
2023-01-11T23:03:39.2797228Z [       OK ] ProcessGroupGlooTest.testAllReduceCPU (288 ms)
2023-01-11T23:03:39.2797794Z [ RUN      ] ProcessGroupGlooTest.testBroadcastCPU
2023-01-11T23:03:39.3221050Z [       OK ] ProcessGroupGlooTest.testBroadcastCPU (42 ms)
2023-01-11T23:03:39.3221576Z [ RUN      ] ProcessGroupGlooTest.testAllToAllCPU
2023-01-11T23:03:39.4657686Z [       OK ] ProcessGroupGlooTest.testAllToAllCPU (143 ms)
2023-01-11T23:03:39.4658198Z [ RUN      ] ProcessGroupGlooTest.testBarrier
2023-01-11T23:03:39.5073402Z [       OK ] ProcessGroupGlooTest.testBarrier (41 ms)
2023-01-11T23:03:39.5073959Z [ RUN      ] ProcessGroupGlooTest.testMonitoredBarrier
2023-01-11T23:03:40.5280264Z [E ProcessGroupGloo.cpp:138] [Rank 0]: Rank 1 failed to pass monitoredBarrier in 1000 ms
2023-01-11T23:03:40.5482914Z [       OK ] ProcessGroupGlooTest.testMonitoredBarrier (1040 ms)
2023-01-11T23:03:40.5483504Z [ RUN      ] ProcessGroupGlooTest.testSequenceNumInit
2023-01-11T23:03:40.7106235Z [       OK ] ProcessGroupGlooTest.testSequenceNumInit (162 ms)
2023-01-11T23:03:40.7106767Z [ RUN      ] ProcessGroupGlooTest.testSend
2023-01-11T23:03:40.7521654Z [       OK ] ProcessGroupGlooTest.testSend (41 ms)
2023-01-11T23:03:40.7522133Z [ RUN      ] ProcessGroupGlooTest.testRecv
2023-01-11T23:03:40.7938033Z [       OK ] ProcessGroupGlooTest.testRecv (41 ms)
2023-01-11T23:03:40.7938555Z [ RUN      ] ProcessGroupGlooTest.testStoreSetGet
2023-01-11T23:03:40.8349943Z [       OK ] ProcessGroupGlooTest.testStoreSetGet (41 ms)
2023-01-11T23:03:40.8350460Z [ RUN      ] ProcessGroupGlooTest.testWaitDelay
2023-01-11T23:03:40.9870885Z [       OK ] ProcessGroupGlooTest.testWaitDelay (152 ms)
2023-01-11T23:03:40.9871691Z [----------] 12 tests from ProcessGroupGlooTest (3095 ms total)
2023-01-11T23:03:40.9871963Z 
2023-01-11T23:03:40.9872207Z [----------] Global test environment tear-down
2023-01-11T23:03:40.9873970Z [==========] 12 tests from 1 test suite ran. (3095 ms total)
2023-01-11T23:03:40.9874331Z [  PASSED  ] 12 tests.
2023-01-11T23:03:41.0727452Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml
2023-01-11T23:03:41.5072614Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2023-01-11T23:03:41.5073262Z [==========] Running 11 tests from 1 test suite.
2023-01-11T23:03:41.5073680Z [----------] Global test environment set-up.
2023-01-11T23:03:41.5074506Z [----------] 11 tests from ProcessGroupNCCLTest
2023-01-11T23:03:41.5074985Z [ RUN      ] ProcessGroupNCCLTest.testAllreduce
2023-01-11T23:03:46.0472899Z [       OK ] ProcessGroupNCCLTest.testAllreduce (4539 ms)
2023-01-11T23:03:46.0473675Z [ RUN      ] ProcessGroupNCCLTest.testBroadcast
2023-01-11T23:03:49.5427688Z [       OK ] ProcessGroupNCCLTest.testBroadcast (3495 ms)
2023-01-11T23:03:49.5428241Z [ RUN      ] ProcessGroupNCCLTest.testReduce
2023-01-11T23:03:53.0382549Z [       OK ] ProcessGroupNCCLTest.testReduce (3495 ms)
2023-01-11T23:03:53.0383098Z [ RUN      ] ProcessGroupNCCLTest.testAllgather
2023-01-11T23:03:54.8234814Z [       OK ] ProcessGroupNCCLTest.testAllgather (1785 ms)
2023-01-11T23:03:54.8235386Z [ RUN      ] ProcessGroupNCCLTest.testAllgatherBase
2023-01-11T23:03:56.5933795Z [       OK ] ProcessGroupNCCLTest.testAllgatherBase (1769 ms)
2023-01-11T23:03:56.5934382Z [ RUN      ] ProcessGroupNCCLTest.testReduceScatter
2023-01-11T23:03:58.3741053Z [       OK ] ProcessGroupNCCLTest.testReduceScatter (1780 ms)
2023-01-11T23:03:58.3741598Z [ RUN      ] ProcessGroupNCCLTest.testSequenceNumInit
2023-01-11T23:03:58.4843011Z [       OK ] ProcessGroupNCCLTest.testSequenceNumInit (110 ms)
2023-01-11T23:03:58.4843680Z [ RUN      ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout
2023-01-11T23:04:01.4887933Z [       OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailTimeout (3004 ms)
2023-01-11T23:04:01.4888744Z [ RUN      ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException
2023-01-11T23:04:04.4904473Z [       OK ] ProcessGroupNCCLTest.testProcessGroupNCCLHealthCheckFailException (3001 ms)
2023-01-11T23:04:04.4905290Z [ RUN      ] ProcessGroupNCCLTest.testReduceScatterBase
2023-01-11T23:04:06.6135221Z [       OK ] ProcessGroupNCCLTest.testReduceScatterBase (2123 ms)
2023-01-11T23:04:06.6135847Z [ RUN      ] ProcessGroupNCCLTest.testBackendName
2023-01-11T23:04:06.6535716Z [       OK ] ProcessGroupNCCLTest.testBackendName (39 ms)
2023-01-11T23:04:06.6536322Z [----------] 11 tests from ProcessGroupNCCLTest (25146 ms total)
2023-01-11T23:04:06.6536588Z 
2023-01-11T23:04:06.6536804Z [----------] Global test environment tear-down
2023-01-11T23:04:06.6537893Z [==========] 11 tests from 1 test suite ran. (25146 ms total)
2023-01-11T23:04:06.6538571Z [  PASSED  ] 11 tests.
2023-01-11T23:04:06.8738255Z + /opt/conda/lib/python3.10/site-packages/torch/bin/ProcessGroupNCCLErrorsTest --gtest_output=xml:test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml
2023-01-11T23:04:07.3128265Z Running main() from /var/lib/jenkins/workspace/third_party/googletest/googletest/src/gtest_main.cc
2023-01-11T23:04:07.3129066Z [==========] Running 3 tests from 1 test suite.
2023-01-11T23:04:07.3129503Z [----------] Global test environment set-up.
2023-01-11T23:04:07.3129969Z [----------] 3 tests from ProcessGroupNCCLErrorsTest
2023-01-11T23:04:07.3130489Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking
2023-01-11T23:04:09.7289154Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsBlocking (2415 ms)
2023-01-11T23:04:09.7289851Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking
2023-01-11T23:04:12.8353723Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLTimedoutErrorsBlocking (3106 ms)
2023-01-11T23:04:12.8354431Z [ RUN      ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking
2023-01-11T23:04:12.9344409Z [       OK ] ProcessGroupNCCLErrorsTest.testNCCLErrorsNonBlocking (98 ms)
2023-01-11T23:04:12.9345292Z [----------] 3 tests from ProcessGroupNCCLErrorsTest (5621 ms total)
2023-01-11T23:04:12.9345568Z 
2023-01-11T23:04:12.9345803Z [----------] Global test environment tear-down
2023-01-11T23:04:12.9346477Z [==========] 3 tests from 1 test suite ran. (5621 ms total)
2023-01-11T23:04:12.9346840Z [  PASSED  ] 3 tests.
2023-01-11T23:04:13.1527298Z + [[ 1 == 1 ]]
2023-01-11T23:04:13.1527989Z + test_rpc
2023-01-11T23:04:13.1528514Z + [[ linux-bionic-cuda11.7-py3.10-gcc7 != *rocm* ]]
2023-01-11T23:04:13.1528899Z + echo 'Testing RPC C++ tests'
2023-01-11T23:04:13.1529158Z Testing RPC C++ tests
2023-01-11T23:04:13.1530542Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cpu.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_cuda_linalg.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_global_deps.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorch_python.so /opt/conda/lib/python3.10/site-packages/torch/lib/libtorchbind_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2023-01-11T23:04:13.1545447Z + ln -sf /opt/conda/lib/python3.10/site-packages/torch/lib/libc10.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10_cuda.so /opt/conda/lib/python3.10/site-packages/torch/lib/libc10d_cuda_test.so /opt/conda/lib/python3.10/site-packages/torch/bin
2023-01-11T23:04:13.1557733Z + ln -sf '/opt/conda/lib/python3.10/site-packages/torch/lib/libtbb*' /opt/conda/lib/python3.10/site-packages/torch/bin
2023-01-11T23:04:13.1568428Z + TEST_REPORTS_DIR=test/test-reports/cpp-rpc/test_rpc
2023-01-11T23:04:13.1568879Z + mkdir -p test/test-reports/cpp-rpc/test_rpc
2023-01-11T23:04:13.1581975Z + /opt/conda/lib/python3.10/site-packages/torch/bin/test_cpp_rpc --gtest_output=xml:test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml
2023-01-11T23:04:13.6160872Z [==========] Running 8 tests from 3 test suites.
2023-01-11T23:04:13.6161350Z [----------] Global test environment set-up.
2023-01-11T23:04:13.6161766Z [----------] 4 tests from WireSerialize
2023-01-11T23:04:13.6162255Z [ RUN      ] WireSerialize.Base
2023-01-11T23:04:13.6175120Z [       OK ] WireSerialize.Base (1 ms)
2023-01-11T23:04:13.6175566Z [ RUN      ] WireSerialize.RecopySparseTensors
2023-01-11T23:04:13.6281830Z [       OK ] WireSerialize.RecopySparseTensors (10 ms)
2023-01-11T23:04:13.6282311Z [ RUN      ] WireSerialize.CloneSparseTensors
2023-01-11T23:04:13.6381451Z [       OK ] WireSerialize.CloneSparseTensors (9 ms)
2023-01-11T23:04:13.6381888Z [ RUN      ] WireSerialize.Errors
2023-01-11T23:04:13.6408764Z [       OK ] WireSerialize.Errors (2 ms)
2023-01-11T23:04:13.6409395Z [----------] 4 tests from WireSerialize (24 ms total)
2023-01-11T23:04:13.6409709Z 
2023-01-11T23:04:13.6409942Z [----------] 1 test from TestE2ETensorPipe
2023-01-11T23:04:13.6410366Z [ RUN      ] TestE2ETensorPipe.TestTrainingLoop
2023-01-11T23:04:14.3579799Z [W tensorpipe_agent.cpp:725] RPC agent for worker encountered error when reading incoming request from worker: pipe closed (this error originated at tensorpipe/core/pipe_impl.cc:356)
2023-01-11T23:04:14.3599018Z [       OK ] TestE2ETensorPipe.TestTrainingLoop (718 ms)
2023-01-11T23:04:14.3599790Z [----------] 1 test from TestE2ETensorPipe (718 ms total)
2023-01-11T23:04:14.3600029Z 
2023-01-11T23:04:14.3600270Z [----------] 3 tests from TensorpipeSerialize
2023-01-11T23:04:14.3600664Z [ RUN      ] TensorpipeSerialize.Base
2023-01-11T23:04:14.3601089Z [       OK ] TensorpipeSerialize.Base (0 ms)
2023-01-11T23:04:14.3601556Z [ RUN      ] TensorpipeSerialize.RecopySparseTensors
2023-01-11T23:04:14.3703902Z [       OK ] TensorpipeSerialize.RecopySparseTensors (10 ms)
2023-01-11T23:04:14.3704829Z [ RUN      ] TensorpipeSerialize.NoDeleterTensors
2023-01-11T23:04:14.3705736Z [       OK ] TensorpipeSerialize.NoDeleterTensors (0 ms)
2023-01-11T23:04:14.3706888Z [----------] 3 tests from TensorpipeSerialize (10 ms total)
2023-01-11T23:04:14.3707332Z 
2023-01-11T23:04:14.3707731Z [----------] Global test environment tear-down
2023-01-11T23:04:14.3709043Z [==========] 8 tests from 3 test suites ran. (754 ms total)
2023-01-11T23:04:14.3710150Z [  PASSED  ] 8 tests.
2023-01-11T23:04:14.3710464Z 
2023-01-11T23:04:14.3710765Z   YOU HAVE 1 DISABLED TEST
2023-01-11T23:04:14.3711080Z 
2023-01-11T23:04:14.4789200Z 
2023-01-11T23:04:14.4827671Z Prepare all required actions
2023-01-11T23:04:14.4828125Z Getting action download info
2023-01-11T23:04:14.7027008Z ##[group]Run ./.github/actions/get-workflow-job-id
2023-01-11T23:04:14.7027290Z with:
2023-01-11T23:04:14.7027739Z   github-token: ***
2023-01-11T23:04:14.7027981Z env:
2023-01-11T23:04:14.7028204Z   GIT_DEFAULT_BRANCH: master
2023-01-11T23:04:14.7028475Z   GPU_FLAG: --gpus all
2023-01-11T23:04:14.7028848Z   DOCKER_CONTAINER_ID: c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56
2023-01-11T23:04:14.7029207Z ##[endgroup]
2023-01-11T23:04:14.7061604Z ##[group]Run nick-fields/retry@3e91a01664abd3c5cd539100d10d33b9c5b68482
2023-01-11T23:04:14.7062180Z with:
2023-01-11T23:04:14.7062402Z   shell: bash
2023-01-11T23:04:14.7062657Z   timeout_minutes: 10
2023-01-11T23:04:14.7062908Z   max_attempts: 5
2023-01-11T23:04:14.7063143Z   retry_wait_seconds: 30
2023-01-11T23:04:14.7063709Z   command: set -eux
python3 -m pip install requests==2.26.0
GHA_WORKFLOW_JOB_ID=$(python3 .github/scripts/get_workflow_job_id.py "${GITHUB_RUN_ID}" "${RUNNER_NAME}")
echo "job-id=${GHA_WORKFLOW_JOB_ID}" >> "${GITHUB_OUTPUT}"

2023-01-11T23:04:14.7064454Z   polling_interval_seconds: 1
2023-01-11T23:04:14.7064768Z   warning_on_retry: true
2023-01-11T23:04:14.7065033Z   continue_on_error: false
2023-01-11T23:04:14.7065261Z env:
2023-01-11T23:04:14.7065499Z   GIT_DEFAULT_BRANCH: master
2023-01-11T23:04:14.7065766Z   GPU_FLAG: --gpus all
2023-01-11T23:04:14.7066136Z   DOCKER_CONTAINER_ID: c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56
2023-01-11T23:04:14.7066619Z   GITHUB_TOKEN: ***
2023-01-11T23:04:14.7066869Z ##[endgroup]
2023-01-11T23:04:14.7736054Z + python3 -m pip install requests==2.26.0
2023-01-11T23:04:15.0655823Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T23:04:15.0884972Z Requirement already satisfied: requests==2.26.0 in /home/ec2-user/.local/lib/python3.7/site-packages (2.26.0)
2023-01-11T23:04:15.1069558Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (3.4)
2023-01-11T23:04:15.1086165Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (1.26.14)
2023-01-11T23:04:15.1308435Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2022.12.7)
2023-01-11T23:04:15.1321154Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests==2.26.0) (2.0.12)
2023-01-11T23:04:15.3790983Z ++ python3 .github/scripts/get_workflow_job_id.py 3896346758 i-0f914c3983ac93cd3
2023-01-11T23:04:18.2640177Z + GHA_WORKFLOW_JOB_ID=10589560299
2023-01-11T23:04:18.2642863Z + echo job-id=10589560299
2023-01-11T23:04:18.7758969Z Command completed after 1 attempt(s).
2023-01-11T23:04:18.7892810Z ##[group]Run kill "$MONITOR_SCRIPT_PID"
2023-01-11T23:04:18.7893176Z kill "$MONITOR_SCRIPT_PID"
2023-01-11T23:04:18.7906201Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T23:04:18.7906514Z env:
2023-01-11T23:04:18.7906770Z   GIT_DEFAULT_BRANCH: master
2023-01-11T23:04:18.7907037Z   GPU_FLAG: --gpus all
2023-01-11T23:04:18.7907420Z   DOCKER_CONTAINER_ID: c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56
2023-01-11T23:04:18.7907803Z   MONITOR_SCRIPT_PID: 56619
2023-01-11T23:04:18.7908057Z ##[endgroup]
2023-01-11T23:04:18.8007798Z Prepare all required actions
2023-01-11T23:04:18.8008174Z Getting action download info
2023-01-11T23:04:18.9711386Z Download action repository 'actions/upload-artifact@v3' (SHA:0b7f8abb1508181956e8e162db84b466c27e18ce)
2023-01-11T23:04:19.1456618Z ##[group]Run ./.github/actions/upload-test-artifacts
2023-01-11T23:04:19.1456921Z with:
2023-01-11T23:04:19.1457260Z   file-suffix: test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589560299
2023-01-11T23:04:19.1457608Z env:
2023-01-11T23:04:19.1457844Z   GIT_DEFAULT_BRANCH: master
2023-01-11T23:04:19.1458091Z   GPU_FLAG: --gpus all
2023-01-11T23:04:19.1458454Z   DOCKER_CONTAINER_ID: c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56
2023-01-11T23:04:19.1458809Z ##[endgroup]
2023-01-11T23:04:19.1489337Z ##[group]Run # Remove any previous test jsons if they exist
2023-01-11T23:04:19.1489700Z # Remove any previous test jsons if they exist
2023-01-11T23:04:19.1490016Z rm -f test-jsons-*.zip
2023-01-11T23:04:19.1490388Z zip -r "test-jsons-${FILE_SUFFIX}.zip" test -i '*.json'
2023-01-11T23:04:19.1502094Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T23:04:19.1502389Z env:
2023-01-11T23:04:19.1502629Z   GIT_DEFAULT_BRANCH: master
2023-01-11T23:04:19.1502889Z   GPU_FLAG: --gpus all
2023-01-11T23:04:19.1503254Z   DOCKER_CONTAINER_ID: c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56
2023-01-11T23:04:19.1503735Z   FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589560299
2023-01-11T23:04:19.1504078Z ##[endgroup]
2023-01-11T23:04:19.1623004Z   adding: test/allowlist_for_publicAPI.json (deflated 78%)
2023-01-11T23:04:19.1657469Z   adding: test/benchmark_utils/callgrind_artifacts.json (deflated 92%)
2023-01-11T23:04:19.1664752Z   adding: test/profiler/profiler_utils_mock_events.json (deflated 87%)
2023-01-11T23:04:19.1666738Z   adding: test/.pytorch-slow-tests.json (deflated 77%)
2023-01-11T23:04:19.1672303Z   adding: test/.pytorch-disabled-tests.json (deflated 84%)
2023-01-11T23:04:19.1696540Z ##[group]Run # Remove any previous test reports if they exist
2023-01-11T23:04:19.1696915Z # Remove any previous test reports if they exist
2023-01-11T23:04:19.1697244Z rm -f test-reports-*.zip
2023-01-11T23:04:19.1697613Z zip -r "test-reports-${FILE_SUFFIX}.zip" test -i '*.xml' -i '*.csv'
2023-01-11T23:04:19.1709383Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T23:04:19.1709683Z env:
2023-01-11T23:04:19.1709929Z   GIT_DEFAULT_BRANCH: master
2023-01-11T23:04:19.1710180Z   GPU_FLAG: --gpus all
2023-01-11T23:04:19.1710552Z   DOCKER_CONTAINER_ID: c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56
2023-01-11T23:04:19.1711032Z   FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589560299
2023-01-11T23:04:19.1711376Z ##[endgroup]
2023-01-11T23:04:19.1862340Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212316.xml (deflated 42%)
2023-01-11T23:04:19.1863288Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212322.xml (deflated 42%)
2023-01-11T23:04:19.1864196Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212329.xml (deflated 44%)
2023-01-11T23:04:19.1865097Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212331.xml (deflated 44%)
2023-01-11T23:04:19.1865979Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212333.xml (deflated 45%)
2023-01-11T23:04:19.1866868Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212335.xml (deflated 45%)
2023-01-11T23:04:19.1867910Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212339.xml (deflated 41%)
2023-01-11T23:04:19.1868832Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212345.xml (deflated 41%)
2023-01-11T23:04:19.1869835Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212351.xml (deflated 44%)
2023-01-11T23:04:19.1870699Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212353.xml (deflated 44%)
2023-01-11T23:04:19.1871595Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212355.xml (deflated 44%)
2023-01-11T23:04:19.1872469Z   adding: test/test-reports/dist-gloo/distributed.algorithms.quantization.test_quantization/TEST-DistQuantizationTests-20230111212357.xml (deflated 45%)
2023-01-11T23:04:19.1873309Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212411.xml (deflated 41%)
2023-01-11T23:04:19.1874096Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212419.xml (deflated 42%)
2023-01-11T23:04:19.1874898Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212422.xml (deflated 42%)
2023-01-11T23:04:19.1875689Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212428.xml (deflated 43%)
2023-01-11T23:04:19.1876465Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212432.xml (deflated 42%)
2023-01-11T23:04:19.1877241Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212439.xml (deflated 41%)
2023-01-11T23:04:19.1878031Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212446.xml (deflated 41%)
2023-01-11T23:04:19.1878812Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212455.xml (deflated 40%)
2023-01-11T23:04:19.1879605Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212502.xml (deflated 41%)
2023-01-11T23:04:19.1880400Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212510.xml (deflated 40%)
2023-01-11T23:04:19.1881160Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212518.xml (deflated 40%)
2023-01-11T23:04:19.1881946Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212526.xml (deflated 40%)
2023-01-11T23:04:19.1882726Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212534.xml (deflated 40%)
2023-01-11T23:04:19.1883518Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212541.xml (deflated 42%)
2023-01-11T23:04:19.1884665Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212546.xml (deflated 41%)
2023-01-11T23:04:19.1885477Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212552.xml (deflated 42%)
2023-01-11T23:04:19.1886249Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212556.xml (deflated 42%)
2023-01-11T23:04:19.1887029Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212604.xml (deflated 42%)
2023-01-11T23:04:19.1887885Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212606.xml (deflated 46%)
2023-01-11T23:04:19.1888685Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212613.xml (deflated 47%)
2023-01-11T23:04:19.1889546Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212620.xml (deflated 49%)
2023-01-11T23:04:19.1890339Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212627.xml (deflated 46%)
2023-01-11T23:04:19.1891098Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212633.xml (deflated 40%)
2023-01-11T23:04:19.1891877Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212640.xml (deflated 41%)
2023-01-11T23:04:19.1892670Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212647.xml (deflated 41%)
2023-01-11T23:04:19.1893447Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212654.xml (deflated 41%)
2023-01-11T23:04:19.1894212Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212700.xml (deflated 41%)
2023-01-11T23:04:19.1894987Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212707.xml (deflated 41%)
2023-01-11T23:04:19.1895767Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212714.xml (deflated 40%)
2023-01-11T23:04:19.1896541Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212721.xml (deflated 41%)
2023-01-11T23:04:19.1897298Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212723.xml (deflated 42%)
2023-01-11T23:04:19.1898094Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212725.xml (deflated 41%)
2023-01-11T23:04:19.1898878Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212732.xml (deflated 42%)
2023-01-11T23:04:19.1899658Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212739.xml (deflated 42%)
2023-01-11T23:04:19.1900433Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212741.xml (deflated 42%)
2023-01-11T23:04:19.1901195Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212744.xml (deflated 42%)
2023-01-11T23:04:19.1901983Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212746.xml (deflated 42%)
2023-01-11T23:04:19.1902767Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212749.xml (deflated 40%)
2023-01-11T23:04:19.1903555Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212755.xml (deflated 40%)
2023-01-11T23:04:19.1904325Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212802.xml (deflated 43%)
2023-01-11T23:04:19.1905104Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212805.xml (deflated 40%)
2023-01-11T23:04:19.1905897Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212811.xml (deflated 41%)
2023-01-11T23:04:19.1906685Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212818.xml (deflated 41%)
2023-01-11T23:04:19.1907510Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212825.xml (deflated 40%)
2023-01-11T23:04:19.1908312Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212832.xml (deflated 42%)
2023-01-11T23:04:19.1909164Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212839.xml (deflated 42%)
2023-01-11T23:04:19.1909947Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212845.xml (deflated 42%)
2023-01-11T23:04:19.1910710Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212852.xml (deflated 42%)
2023-01-11T23:04:19.1911496Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212859.xml (deflated 40%)
2023-01-11T23:04:19.1912287Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212905.xml (deflated 40%)
2023-01-11T23:04:19.1913068Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212912.xml (deflated 41%)
2023-01-11T23:04:19.1913857Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212919.xml (deflated 41%)
2023-01-11T23:04:19.1914620Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212925.xml (deflated 40%)
2023-01-11T23:04:19.1915408Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212932.xml (deflated 40%)
2023-01-11T23:04:19.1916184Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212939.xml (deflated 40%)
2023-01-11T23:04:19.1916960Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212946.xml (deflated 40%)
2023-01-11T23:04:19.1917727Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212953.xml (deflated 41%)
2023-01-11T23:04:19.1918512Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111212959.xml (deflated 40%)
2023-01-11T23:04:19.1919293Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213006.xml (deflated 41%)
2023-01-11T23:04:19.1920072Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213013.xml (deflated 42%)
2023-01-11T23:04:19.1920833Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213019.xml (deflated 42%)
2023-01-11T23:04:19.1921629Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213026.xml (deflated 42%)
2023-01-11T23:04:19.1922406Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213033.xml (deflated 41%)
2023-01-11T23:04:19.1923176Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213040.xml (deflated 41%)
2023-01-11T23:04:19.1923937Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213046.xml (deflated 40%)
2023-01-11T23:04:19.1925098Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213054.xml (deflated 41%)
2023-01-11T23:04:19.1925883Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213102.xml (deflated 41%)
2023-01-11T23:04:19.1926656Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213109.xml (deflated 41%)
2023-01-11T23:04:19.1927529Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213117.xml (deflated 41%)
2023-01-11T23:04:19.1928320Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213123.xml (deflated 41%)
2023-01-11T23:04:19.1929181Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213130.xml (deflated 41%)
2023-01-11T23:04:19.1929961Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213137.xml (deflated 41%)
2023-01-11T23:04:19.1930737Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213146.xml (deflated 41%)
2023-01-11T23:04:19.1931502Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213154.xml (deflated 41%)
2023-01-11T23:04:19.1932284Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213203.xml (deflated 42%)
2023-01-11T23:04:19.1933062Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213205.xml (deflated 42%)
2023-01-11T23:04:19.1933849Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213208.xml (deflated 42%)
2023-01-11T23:04:19.1934610Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213210.xml (deflated 42%)
2023-01-11T23:04:19.1935393Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213212.xml (deflated 43%)
2023-01-11T23:04:19.1936171Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213215.xml (deflated 42%)
2023-01-11T23:04:19.1936949Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213217.xml (deflated 43%)
2023-01-11T23:04:19.1937712Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213220.xml (deflated 42%)
2023-01-11T23:04:19.1938503Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213222.xml (deflated 43%)
2023-01-11T23:04:19.1939278Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213224.xml (deflated 42%)
2023-01-11T23:04:19.1940055Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213227.xml (deflated 43%)
2023-01-11T23:04:19.1940834Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213229.xml (deflated 43%)
2023-01-11T23:04:19.1941596Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213231.xml (deflated 42%)
2023-01-11T23:04:19.1942376Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213234.xml (deflated 42%)
2023-01-11T23:04:19.1943153Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213236.xml (deflated 42%)
2023-01-11T23:04:19.1943931Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213239.xml (deflated 43%)
2023-01-11T23:04:19.1944688Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213241.xml (deflated 42%)
2023-01-11T23:04:19.1945469Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213243.xml (deflated 42%)
2023-01-11T23:04:19.1946317Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213246.xml (deflated 42%)
2023-01-11T23:04:19.1947109Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213248.xml (deflated 42%)
2023-01-11T23:04:19.1947863Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213251.xml (deflated 42%)
2023-01-11T23:04:19.1948716Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213253.xml (deflated 42%)
2023-01-11T23:04:19.1949493Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213255.xml (deflated 42%)
2023-01-11T23:04:19.1950274Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213258.xml (deflated 43%)
2023-01-11T23:04:19.1951048Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213300.xml (deflated 41%)
2023-01-11T23:04:19.1951820Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213308.xml (deflated 42%)
2023-01-11T23:04:19.1952863Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213315.xml (deflated 42%)
2023-01-11T23:04:19.1953657Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213318.xml (deflated 41%)
2023-01-11T23:04:19.1954435Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213325.xml (deflated 41%)
2023-01-11T23:04:19.1955203Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213333.xml (deflated 40%)
2023-01-11T23:04:19.1955998Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213341.xml (deflated 42%)
2023-01-11T23:04:19.1956785Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213348.xml (deflated 42%)
2023-01-11T23:04:19.1957578Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213354.xml (deflated 42%)
2023-01-11T23:04:19.1958360Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213401.xml (deflated 40%)
2023-01-11T23:04:19.1959139Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213409.xml (deflated 42%)
2023-01-11T23:04:19.1959922Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213411.xml (deflated 42%)
2023-01-11T23:04:19.1960710Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213418.xml (deflated 41%)
2023-01-11T23:04:19.1961504Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213425.xml (deflated 41%)
2023-01-11T23:04:19.1962276Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213432.xml (deflated 41%)
2023-01-11T23:04:19.1963066Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213434.xml (deflated 41%)
2023-01-11T23:04:19.1963849Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213436.xml (deflated 40%)
2023-01-11T23:04:19.1964857Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213439.xml (deflated 41%)
2023-01-11T23:04:19.1965644Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213441.xml (deflated 41%)
2023-01-11T23:04:19.1966522Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213443.xml (deflated 40%)
2023-01-11T23:04:19.1968355Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213446.xml (deflated 41%)
2023-01-11T23:04:19.1970291Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213448.xml (deflated 41%)
2023-01-11T23:04:19.1971926Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213451.xml (deflated 41%)
2023-01-11T23:04:19.1973542Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213457.xml (deflated 42%)
2023-01-11T23:04:19.1975202Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213502.xml (deflated 40%)
2023-01-11T23:04:19.1976866Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213508.xml (deflated 42%)
2023-01-11T23:04:19.1978507Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213515.xml (deflated 41%)
2023-01-11T23:04:19.1980139Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213523.xml (deflated 42%)
2023-01-11T23:04:19.1981756Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213527.xml (deflated 42%)
2023-01-11T23:04:19.1983394Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213531.xml (deflated 42%)
2023-01-11T23:04:19.1985001Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213535.xml (deflated 41%)
2023-01-11T23:04:19.1986635Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213545.xml (deflated 40%)
2023-01-11T23:04:19.1988289Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213553.xml (deflated 40%)
2023-01-11T23:04:19.1989897Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213602.xml (deflated 40%)
2023-01-11T23:04:19.1991542Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213610.xml (deflated 40%)
2023-01-11T23:04:19.1993178Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213618.xml (deflated 42%)
2023-01-11T23:04:19.1994784Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213622.xml (deflated 42%)
2023-01-11T23:04:19.1996390Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213627.xml (deflated 40%)
2023-01-11T23:04:19.1998043Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213634.xml (deflated 40%)
2023-01-11T23:04:19.1999708Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213642.xml (deflated 40%)
2023-01-11T23:04:19.2001397Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213650.xml (deflated 40%)
2023-01-11T23:04:19.2003016Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213658.xml (deflated 42%)
2023-01-11T23:04:19.2004932Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213702.xml (deflated 41%)
2023-01-11T23:04:19.2006543Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213709.xml (deflated 42%)
2023-01-11T23:04:19.2008317Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213713.xml (deflated 41%)
2023-01-11T23:04:19.2009985Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213721.xml (deflated 42%)
2023-01-11T23:04:19.2011786Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213725.xml (deflated 42%)
2023-01-11T23:04:19.2013444Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213729.xml (deflated 40%)
2023-01-11T23:04:19.2015120Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213738.xml (deflated 40%)
2023-01-11T23:04:19.2016784Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213747.xml (deflated 42%)
2023-01-11T23:04:19.2018408Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213751.xml (deflated 40%)
2023-01-11T23:04:19.2020077Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213759.xml (deflated 41%)
2023-01-11T23:04:19.2021757Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213807.xml (deflated 41%)
2023-01-11T23:04:19.2023403Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213816.xml (deflated 41%)
2023-01-11T23:04:19.2025065Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213824.xml (deflated 40%)
2023-01-11T23:04:19.2026728Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213832.xml (deflated 41%)
2023-01-11T23:04:19.2028373Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213841.xml (deflated 40%)
2023-01-11T23:04:19.2030005Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213849.xml (deflated 41%)
2023-01-11T23:04:19.2031614Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213857.xml (deflated 41%)
2023-01-11T23:04:19.2033242Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213906.xml (deflated 41%)
2023-01-11T23:04:19.2034884Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213914.xml (deflated 41%)
2023-01-11T23:04:19.2036525Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213922.xml (deflated 41%)
2023-01-11T23:04:19.2038125Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213930.xml (deflated 41%)
2023-01-11T23:04:19.2039814Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213939.xml (deflated 42%)
2023-01-11T23:04:19.2041482Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213943.xml (deflated 40%)
2023-01-11T23:04:19.2043128Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213951.xml (deflated 40%)
2023-01-11T23:04:19.2044970Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111213959.xml (deflated 40%)
2023-01-11T23:04:19.2046570Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214006.xml (deflated 41%)
2023-01-11T23:04:19.2048346Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214014.xml (deflated 40%)
2023-01-11T23:04:19.2049963Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214022.xml (deflated 40%)
2023-01-11T23:04:19.2051579Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214040.xml (deflated 41%)
2023-01-11T23:04:19.2053353Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214048.xml (deflated 41%)
2023-01-11T23:04:19.2055018Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214056.xml (deflated 41%)
2023-01-11T23:04:19.2056664Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214104.xml (deflated 40%)
2023-01-11T23:04:19.2058342Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214112.xml (deflated 42%)
2023-01-11T23:04:19.2059999Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214116.xml (deflated 42%)
2023-01-11T23:04:19.2061674Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214120.xml (deflated 42%)
2023-01-11T23:04:19.2063324Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214129.xml (deflated 41%)
2023-01-11T23:04:19.2064969Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214136.xml (deflated 42%)
2023-01-11T23:04:19.2066560Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214141.xml (deflated 41%)
2023-01-11T23:04:19.2068171Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214149.xml (deflated 42%)
2023-01-11T23:04:19.2069792Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214153.xml (deflated 41%)
2023-01-11T23:04:19.2071438Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214202.xml (deflated 41%)
2023-01-11T23:04:19.2073088Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214209.xml (deflated 40%)
2023-01-11T23:04:19.2074759Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214217.xml (deflated 42%)
2023-01-11T23:04:19.2076397Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214221.xml (deflated 42%)
2023-01-11T23:04:19.2078021Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214225.xml (deflated 42%)
2023-01-11T23:04:19.2079622Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214229.xml (deflated 40%)
2023-01-11T23:04:19.2081275Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214237.xml (deflated 41%)
2023-01-11T23:04:19.2082929Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214245.xml (deflated 41%)
2023-01-11T23:04:19.2084814Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214252.xml (deflated 41%)
2023-01-11T23:04:19.2086421Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214258.xml (deflated 42%)
2023-01-11T23:04:19.2088054Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214303.xml (deflated 42%)
2023-01-11T23:04:19.2089798Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214307.xml (deflated 40%)
2023-01-11T23:04:19.2091462Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214313.xml (deflated 41%)
2023-01-11T23:04:19.2093253Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214320.xml (deflated 41%)
2023-01-11T23:04:19.2094889Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214327.xml (deflated 42%)
2023-01-11T23:04:19.2096519Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214329.xml (deflated 40%)
2023-01-11T23:04:19.2098181Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214336.xml (deflated 42%)
2023-01-11T23:04:19.2099822Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214343.xml (deflated 40%)
2023-01-11T23:04:19.2101488Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214349.xml (deflated 42%)
2023-01-11T23:04:19.2103135Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214354.xml (deflated 41%)
2023-01-11T23:04:19.2104742Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214400.xml (deflated 41%)
2023-01-11T23:04:19.2106332Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214407.xml (deflated 41%)
2023-01-11T23:04:19.2107957Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214414.xml (deflated 40%)
2023-01-11T23:04:19.2109586Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214421.xml (deflated 41%)
2023-01-11T23:04:19.2111184Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214428.xml (deflated 40%)
2023-01-11T23:04:19.2112781Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214436.xml (deflated 40%)
2023-01-11T23:04:19.2114387Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214443.xml (deflated 41%)
2023-01-11T23:04:19.2116011Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214450.xml (deflated 41%)
2023-01-11T23:04:19.2117744Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214456.xml (deflated 41%)
2023-01-11T23:04:19.2119407Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214503.xml (deflated 41%)
2023-01-11T23:04:19.2121033Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214527.xml (deflated 40%)
2023-01-11T23:04:19.2122638Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214550.xml (deflated 42%)
2023-01-11T23:04:19.2124458Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214557.xml (deflated 40%)
2023-01-11T23:04:19.2126134Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214606.xml (deflated 40%)
2023-01-11T23:04:19.2127780Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214613.xml (deflated 40%)
2023-01-11T23:04:19.2129414Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214620.xml (deflated 41%)
2023-01-11T23:04:19.2131214Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214627.xml (deflated 42%)
2023-01-11T23:04:19.2132839Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214629.xml (deflated 42%)
2023-01-11T23:04:19.2134621Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214631.xml (deflated 42%)
2023-01-11T23:04:19.2136252Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214634.xml (deflated 42%)
2023-01-11T23:04:19.2137905Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214636.xml (deflated 42%)
2023-01-11T23:04:19.2139506Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214639.xml (deflated 42%)
2023-01-11T23:04:19.2141143Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214641.xml (deflated 42%)
2023-01-11T23:04:19.2142774Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214643.xml (deflated 41%)
2023-01-11T23:04:19.2144439Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214646.xml (deflated 40%)
2023-01-11T23:04:19.2146106Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214653.xml (deflated 40%)
2023-01-11T23:04:19.2147707Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214659.xml (deflated 41%)
2023-01-11T23:04:19.2149295Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214702.xml (deflated 42%)
2023-01-11T23:04:19.2150968Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214704.xml (deflated 42%)
2023-01-11T23:04:19.2152618Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214708.xml (deflated 40%)
2023-01-11T23:04:19.2154232Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214716.xml (deflated 41%)
2023-01-11T23:04:19.2155829Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214724.xml (deflated 40%)
2023-01-11T23:04:19.2157452Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214733.xml (deflated 42%)
2023-01-11T23:04:19.2159073Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214737.xml (deflated 41%)
2023-01-11T23:04:19.2160721Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214741.xml (deflated 41%)
2023-01-11T23:04:19.2162336Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214748.xml (deflated 40%)
2023-01-11T23:04:19.2164001Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214754.xml (deflated 41%)
2023-01-11T23:04:19.2165896Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214758.xml (deflated 40%)
2023-01-11T23:04:19.2167565Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214805.xml (deflated 41%)
2023-01-11T23:04:19.2169196Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214812.xml (deflated 41%)
2023-01-11T23:04:19.2170992Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214819.xml (deflated 40%)
2023-01-11T23:04:19.2172622Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214826.xml (deflated 42%)
2023-01-11T23:04:19.2174250Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214832.xml (deflated 42%)
2023-01-11T23:04:19.2176028Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214839.xml (deflated 42%)
2023-01-11T23:04:19.2177699Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214846.xml (deflated 42%)
2023-01-11T23:04:19.2179345Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214852.xml (deflated 41%)
2023-01-11T23:04:19.2181056Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214859.xml (deflated 41%)
2023-01-11T23:04:19.2182698Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214906.xml (deflated 42%)
2023-01-11T23:04:19.2184314Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214908.xml (deflated 41%)
2023-01-11T23:04:19.2185977Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214915.xml (deflated 43%)
2023-01-11T23:04:19.2187631Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214918.xml (deflated 43%)
2023-01-11T23:04:19.2189294Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214920.xml (deflated 41%)
2023-01-11T23:04:19.2190902Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214927.xml (deflated 42%)
2023-01-11T23:04:19.2192555Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214929.xml (deflated 42%)
2023-01-11T23:04:19.2194187Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214932.xml (deflated 41%)
2023-01-11T23:04:19.2195854Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214938.xml (deflated 41%)
2023-01-11T23:04:19.2197478Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214945.xml (deflated 40%)
2023-01-11T23:04:19.2199112Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214952.xml (deflated 40%)
2023-01-11T23:04:19.2200735Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111214959.xml (deflated 41%)
2023-01-11T23:04:19.2202394Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215001.xml (deflated 42%)
2023-01-11T23:04:19.2203977Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215003.xml (deflated 40%)
2023-01-11T23:04:19.2205955Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215010.xml (deflated 42%)
2023-01-11T23:04:19.2207608Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215017.xml (deflated 41%)
2023-01-11T23:04:19.2209270Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215023.xml (deflated 41%)
2023-01-11T23:04:19.2210885Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215030.xml (deflated 41%)
2023-01-11T23:04:19.2212692Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215037.xml (deflated 40%)
2023-01-11T23:04:19.2214361Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215044.xml (deflated 40%)
2023-01-11T23:04:19.2216149Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215050.xml (deflated 41%)
2023-01-11T23:04:19.2217730Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215057.xml (deflated 41%)
2023-01-11T23:04:19.2219330Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215100.xml (deflated 41%)
2023-01-11T23:04:19.2220949Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215102.xml (deflated 41%)
2023-01-11T23:04:19.2222602Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215104.xml (deflated 41%)
2023-01-11T23:04:19.2224244Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215111.xml (deflated 41%)
2023-01-11T23:04:19.2225977Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215118.xml (deflated 41%)
2023-01-11T23:04:19.2227615Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215125.xml (deflated 41%)
2023-01-11T23:04:19.2229235Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215131.xml (deflated 41%)
2023-01-11T23:04:19.2230844Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215138.xml (deflated 40%)
2023-01-11T23:04:19.2232521Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215146.xml (deflated 41%)
2023-01-11T23:04:19.2234158Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215154.xml (deflated 41%)
2023-01-11T23:04:19.2235813Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215201.xml (deflated 40%)
2023-01-11T23:04:19.2237448Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215208.xml (deflated 41%)
2023-01-11T23:04:19.2239106Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215216.xml (deflated 40%)
2023-01-11T23:04:19.2240778Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215229.xml (deflated 40%)
2023-01-11T23:04:19.2242444Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215244.xml (deflated 41%)
2023-01-11T23:04:19.2244083Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215252.xml (deflated 42%)
2023-01-11T23:04:19.2246035Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215255.xml (deflated 42%)
2023-01-11T23:04:19.2247729Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215301.xml (deflated 42%)
2023-01-11T23:04:19.2249380Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215306.xml (deflated 42%)
2023-01-11T23:04:19.2251015Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215312.xml (deflated 41%)
2023-01-11T23:04:19.2252620Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215319.xml (deflated 40%)
2023-01-11T23:04:19.2254395Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215328.xml (deflated 40%)
2023-01-11T23:04:19.2256045Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215335.xml (deflated 41%)
2023-01-11T23:04:19.2257839Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215343.xml (deflated 39%)
2023-01-11T23:04:19.2259446Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215351.xml (deflated 40%)
2023-01-11T23:04:19.2261089Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215400.xml (deflated 40%)
2023-01-11T23:04:19.2262747Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215408.xml (deflated 40%)
2023-01-11T23:04:19.2264395Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215416.xml (deflated 42%)
2023-01-11T23:04:19.2265996Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215420.xml (deflated 41%)
2023-01-11T23:04:19.2267639Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215426.xml (deflated 42%)
2023-01-11T23:04:19.2269233Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215431.xml (deflated 42%)
2023-01-11T23:04:19.2270846Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215438.xml (deflated 42%)
2023-01-11T23:04:19.2272445Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215441.xml (deflated 46%)
2023-01-11T23:04:19.2274103Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215448.xml (deflated 47%)
2023-01-11T23:04:19.2275774Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215454.xml (deflated 48%)
2023-01-11T23:04:19.2277438Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215501.xml (deflated 46%)
2023-01-11T23:04:19.2279068Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215508.xml (deflated 40%)
2023-01-11T23:04:19.2280759Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215515.xml (deflated 41%)
2023-01-11T23:04:19.2282445Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215521.xml (deflated 41%)
2023-01-11T23:04:19.2284472Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215528.xml (deflated 42%)
2023-01-11T23:04:19.2286116Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215535.xml (deflated 41%)
2023-01-11T23:04:19.2287752Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215542.xml (deflated 41%)
2023-01-11T23:04:19.2289355Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215548.xml (deflated 40%)
2023-01-11T23:04:19.2290986Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215555.xml (deflated 41%)
2023-01-11T23:04:19.2292645Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215558.xml (deflated 41%)
2023-01-11T23:04:19.2294447Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215600.xml (deflated 41%)
2023-01-11T23:04:19.2296065Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215607.xml (deflated 42%)
2023-01-11T23:04:19.2297681Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215614.xml (deflated 42%)
2023-01-11T23:04:19.2299487Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215616.xml (deflated 42%)
2023-01-11T23:04:19.2301139Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215618.xml (deflated 42%)
2023-01-11T23:04:19.2302748Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215621.xml (deflated 42%)
2023-01-11T23:04:19.2304368Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215623.xml (deflated 41%)
2023-01-11T23:04:19.2305996Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215630.xml (deflated 40%)
2023-01-11T23:04:19.2307656Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215637.xml (deflated 43%)
2023-01-11T23:04:19.2309330Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215639.xml (deflated 40%)
2023-01-11T23:04:19.2310957Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215646.xml (deflated 40%)
2023-01-11T23:04:19.2312571Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215653.xml (deflated 40%)
2023-01-11T23:04:19.2314207Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215700.xml (deflated 40%)
2023-01-11T23:04:19.2315878Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215706.xml (deflated 42%)
2023-01-11T23:04:19.2317495Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215713.xml (deflated 41%)
2023-01-11T23:04:19.2319177Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215720.xml (deflated 42%)
2023-01-11T23:04:19.2320848Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215727.xml (deflated 42%)
2023-01-11T23:04:19.2322479Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215734.xml (deflated 40%)
2023-01-11T23:04:19.2324072Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215740.xml (deflated 40%)
2023-01-11T23:04:19.2325995Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215747.xml (deflated 41%)
2023-01-11T23:04:19.2327646Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215754.xml (deflated 41%)
2023-01-11T23:04:19.2329277Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215801.xml (deflated 41%)
2023-01-11T23:04:19.2330913Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215807.xml (deflated 41%)
2023-01-11T23:04:19.2332573Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215814.xml (deflated 40%)
2023-01-11T23:04:19.2334234Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215821.xml (deflated 41%)
2023-01-11T23:04:19.2336084Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215828.xml (deflated 41%)
2023-01-11T23:04:19.2337745Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215835.xml (deflated 40%)
2023-01-11T23:04:19.2339583Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215841.xml (deflated 42%)
2023-01-11T23:04:19.2341171Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215848.xml (deflated 42%)
2023-01-11T23:04:19.2342811Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215855.xml (deflated 41%)
2023-01-11T23:04:19.2344429Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215901.xml (deflated 42%)
2023-01-11T23:04:19.2346064Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215908.xml (deflated 41%)
2023-01-11T23:04:19.2347702Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215915.xml (deflated 40%)
2023-01-11T23:04:19.2349347Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215922.xml (deflated 40%)
2023-01-11T23:04:19.2350961Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215930.xml (deflated 41%)
2023-01-11T23:04:19.2352580Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215937.xml (deflated 40%)
2023-01-11T23:04:19.2354242Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215944.xml (deflated 41%)
2023-01-11T23:04:19.2355872Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215952.xml (deflated 40%)
2023-01-11T23:04:19.2357477Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111215959.xml (deflated 41%)
2023-01-11T23:04:19.2359105Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220005.xml (deflated 40%)
2023-01-11T23:04:19.2360727Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220012.xml (deflated 41%)
2023-01-11T23:04:19.2362363Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220021.xml (deflated 41%)
2023-01-11T23:04:19.2363890Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220029.xml (deflated 41%)
2023-01-11T23:04:19.2365772Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220038.xml (deflated 42%)
2023-01-11T23:04:19.2367342Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220040.xml (deflated 42%)
2023-01-11T23:04:19.2368916Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220043.xml (deflated 42%)
2023-01-11T23:04:19.2370502Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220045.xml (deflated 41%)
2023-01-11T23:04:19.2372143Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220048.xml (deflated 43%)
2023-01-11T23:04:19.2373785Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220050.xml (deflated 42%)
2023-01-11T23:04:19.2375439Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220052.xml (deflated 43%)
2023-01-11T23:04:19.2377211Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220055.xml (deflated 42%)
2023-01-11T23:04:19.2378842Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220057.xml (deflated 42%)
2023-01-11T23:04:19.2380577Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220100.xml (deflated 42%)
2023-01-11T23:04:19.2382176Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220102.xml (deflated 43%)
2023-01-11T23:04:19.2383730Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220104.xml (deflated 43%)
2023-01-11T23:04:19.2385369Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220107.xml (deflated 42%)
2023-01-11T23:04:19.2387013Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220109.xml (deflated 42%)
2023-01-11T23:04:19.2388637Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220111.xml (deflated 42%)
2023-01-11T23:04:19.2390254Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220114.xml (deflated 43%)
2023-01-11T23:04:19.2391896Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220116.xml (deflated 42%)
2023-01-11T23:04:19.2393557Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220119.xml (deflated 42%)
2023-01-11T23:04:19.2395227Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220121.xml (deflated 43%)
2023-01-11T23:04:19.2396873Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220123.xml (deflated 42%)
2023-01-11T23:04:19.2398472Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220126.xml (deflated 42%)
2023-01-11T23:04:19.2400096Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220128.xml (deflated 42%)
2023-01-11T23:04:19.2401705Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220131.xml (deflated 42%)
2023-01-11T23:04:19.2403319Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220133.xml (deflated 43%)
2023-01-11T23:04:19.2405115Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220135.xml (deflated 41%)
2023-01-11T23:04:19.2406785Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220144.xml (deflated 42%)
2023-01-11T23:04:19.2408447Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220150.xml (deflated 42%)
2023-01-11T23:04:19.2410083Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220153.xml (deflated 41%)
2023-01-11T23:04:19.2411679Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220200.xml (deflated 41%)
2023-01-11T23:04:19.2413306Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220209.xml (deflated 41%)
2023-01-11T23:04:19.2414926Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220216.xml (deflated 42%)
2023-01-11T23:04:19.2416723Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220223.xml (deflated 42%)
2023-01-11T23:04:19.2418357Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220230.xml (deflated 42%)
2023-01-11T23:04:19.2420015Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220236.xml (deflated 41%)
2023-01-11T23:04:19.2421812Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220244.xml (deflated 41%)
2023-01-11T23:04:19.2423451Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220252.xml (deflated 42%)
2023-01-11T23:04:19.2425119Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220259.xml (deflated 41%)
2023-01-11T23:04:19.2426850Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220305.xml (deflated 41%)
2023-01-11T23:04:19.2428486Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220312.xml (deflated 41%)
2023-01-11T23:04:19.2430126Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220315.xml (deflated 41%)
2023-01-11T23:04:19.2431760Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220317.xml (deflated 40%)
2023-01-11T23:04:19.2433419Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220319.xml (deflated 41%)
2023-01-11T23:04:19.2435052Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220322.xml (deflated 41%)
2023-01-11T23:04:19.2436669Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220324.xml (deflated 41%)
2023-01-11T23:04:19.2438275Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220327.xml (deflated 41%)
2023-01-11T23:04:19.2439925Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220329.xml (deflated 41%)
2023-01-11T23:04:19.2441585Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220331.xml (deflated 41%)
2023-01-11T23:04:19.2443220Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220338.xml (deflated 42%)
2023-01-11T23:04:19.2445027Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220342.xml (deflated 40%)
2023-01-11T23:04:19.2446644Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220349.xml (deflated 42%)
2023-01-11T23:04:19.2448286Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220356.xml (deflated 41%)
2023-01-11T23:04:19.2449925Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220403.xml (deflated 42%)
2023-01-11T23:04:19.2451571Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220408.xml (deflated 42%)
2023-01-11T23:04:19.2453239Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220412.xml (deflated 42%)
2023-01-11T23:04:19.2454894Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220416.xml (deflated 41%)
2023-01-11T23:04:19.2456523Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220425.xml (deflated 40%)
2023-01-11T23:04:19.2458270Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220434.xml (deflated 40%)
2023-01-11T23:04:19.2459917Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220443.xml (deflated 40%)
2023-01-11T23:04:19.2461730Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220451.xml (deflated 40%)
2023-01-11T23:04:19.2463361Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220459.xml (deflated 42%)
2023-01-11T23:04:19.2464944Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220503.xml (deflated 42%)
2023-01-11T23:04:19.2466557Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220507.xml (deflated 40%)
2023-01-11T23:04:19.2468177Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220515.xml (deflated 40%)
2023-01-11T23:04:19.2469840Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220523.xml (deflated 40%)
2023-01-11T23:04:19.2471477Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220531.xml (deflated 40%)
2023-01-11T23:04:19.2473058Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220539.xml (deflated 42%)
2023-01-11T23:04:19.2474678Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220543.xml (deflated 41%)
2023-01-11T23:04:19.2476251Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220550.xml (deflated 42%)
2023-01-11T23:04:19.2477866Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220554.xml (deflated 41%)
2023-01-11T23:04:19.2479363Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220602.xml (deflated 42%)
2023-01-11T23:04:19.2480229Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220606.xml (deflated 42%)
2023-01-11T23:04:19.2481093Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220610.xml (deflated 40%)
2023-01-11T23:04:19.2481895Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220618.xml (deflated 40%)
2023-01-11T23:04:19.2482661Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220627.xml (deflated 42%)
2023-01-11T23:04:19.2483451Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220632.xml (deflated 40%)
2023-01-11T23:04:19.2484518Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220640.xml (deflated 41%)
2023-01-11T23:04:19.2485442Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220648.xml (deflated 41%)
2023-01-11T23:04:19.2486217Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220656.xml (deflated 41%)
2023-01-11T23:04:19.2486998Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220705.xml (deflated 41%)
2023-01-11T23:04:19.2487778Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220713.xml (deflated 40%)
2023-01-11T23:04:19.2488556Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220722.xml (deflated 41%)
2023-01-11T23:04:19.2489460Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220730.xml (deflated 41%)
2023-01-11T23:04:19.2490267Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220738.xml (deflated 40%)
2023-01-11T23:04:19.2491128Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220747.xml (deflated 41%)
2023-01-11T23:04:19.2491910Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220755.xml (deflated 41%)
2023-01-11T23:04:19.2492674Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220803.xml (deflated 41%)
2023-01-11T23:04:19.2493448Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220812.xml (deflated 41%)
2023-01-11T23:04:19.2494229Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220820.xml (deflated 42%)
2023-01-11T23:04:19.2495005Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220824.xml (deflated 41%)
2023-01-11T23:04:19.2495770Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220832.xml (deflated 40%)
2023-01-11T23:04:19.2496552Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220840.xml (deflated 40%)
2023-01-11T23:04:19.2497328Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220847.xml (deflated 40%)
2023-01-11T23:04:19.2498103Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220855.xml (deflated 40%)
2023-01-11T23:04:19.2498865Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220903.xml (deflated 40%)
2023-01-11T23:04:19.2499640Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220921.xml (deflated 41%)
2023-01-11T23:04:19.2500421Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220929.xml (deflated 41%)
2023-01-11T23:04:19.2501191Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220937.xml (deflated 41%)
2023-01-11T23:04:19.2501956Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220945.xml (deflated 41%)
2023-01-11T23:04:19.2502734Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220953.xml (deflated 42%)
2023-01-11T23:04:19.2503509Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111220957.xml (deflated 42%)
2023-01-11T23:04:19.2504288Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221002.xml (deflated 42%)
2023-01-11T23:04:19.2505052Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221010.xml (deflated 41%)
2023-01-11T23:04:19.2505825Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221018.xml (deflated 42%)
2023-01-11T23:04:19.2506598Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221022.xml (deflated 41%)
2023-01-11T23:04:19.2507370Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221030.xml (deflated 42%)
2023-01-11T23:04:19.2508207Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221034.xml (deflated 41%)
2023-01-11T23:04:19.2508998Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221043.xml (deflated 41%)
2023-01-11T23:04:19.2509772Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221050.xml (deflated 41%)
2023-01-11T23:04:19.2510609Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221058.xml (deflated 42%)
2023-01-11T23:04:19.2511367Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221102.xml (deflated 42%)
2023-01-11T23:04:19.2512146Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221106.xml (deflated 42%)
2023-01-11T23:04:19.2512919Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221110.xml (deflated 41%)
2023-01-11T23:04:19.2513700Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221119.xml (deflated 41%)
2023-01-11T23:04:19.2514460Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221126.xml (deflated 41%)
2023-01-11T23:04:19.2515244Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221133.xml (deflated 41%)
2023-01-11T23:04:19.2516014Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221140.xml (deflated 42%)
2023-01-11T23:04:19.2516787Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221144.xml (deflated 42%)
2023-01-11T23:04:19.2517540Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221148.xml (deflated 40%)
2023-01-11T23:04:19.2518318Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221155.xml (deflated 41%)
2023-01-11T23:04:19.2519095Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221202.xml (deflated 41%)
2023-01-11T23:04:19.2519874Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221209.xml (deflated 42%)
2023-01-11T23:04:19.2520642Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221211.xml (deflated 41%)
2023-01-11T23:04:19.2521401Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221218.xml (deflated 42%)
2023-01-11T23:04:19.2522171Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221225.xml (deflated 40%)
2023-01-11T23:04:19.2522947Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221231.xml (deflated 42%)
2023-01-11T23:04:19.2523721Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221236.xml (deflated 41%)
2023-01-11T23:04:19.2524972Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221242.xml (deflated 41%)
2023-01-11T23:04:19.2525819Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221249.xml (deflated 41%)
2023-01-11T23:04:19.2526596Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221256.xml (deflated 40%)
2023-01-11T23:04:19.2527372Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221303.xml (deflated 40%)
2023-01-11T23:04:19.2528233Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221309.xml (deflated 40%)
2023-01-11T23:04:19.2529031Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221318.xml (deflated 41%)
2023-01-11T23:04:19.2529892Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221324.xml (deflated 41%)
2023-01-11T23:04:19.2530669Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221331.xml (deflated 41%)
2023-01-11T23:04:19.2531424Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221338.xml (deflated 41%)
2023-01-11T23:04:19.2532197Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221345.xml (deflated 41%)
2023-01-11T23:04:19.2532974Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221408.xml (deflated 41%)
2023-01-11T23:04:19.2533749Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221432.xml (deflated 42%)
2023-01-11T23:04:19.2534511Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221439.xml (deflated 40%)
2023-01-11T23:04:19.2535285Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221448.xml (deflated 41%)
2023-01-11T23:04:19.2536057Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221454.xml (deflated 40%)
2023-01-11T23:04:19.2536829Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221501.xml (deflated 41%)
2023-01-11T23:04:19.2537596Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221508.xml (deflated 42%)
2023-01-11T23:04:19.2538376Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221511.xml (deflated 42%)
2023-01-11T23:04:19.2539148Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221513.xml (deflated 42%)
2023-01-11T23:04:19.2539930Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221515.xml (deflated 42%)
2023-01-11T23:04:19.2540689Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221518.xml (deflated 42%)
2023-01-11T23:04:19.2541457Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221520.xml (deflated 42%)
2023-01-11T23:04:19.2542230Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221523.xml (deflated 42%)
2023-01-11T23:04:19.2543005Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221525.xml (deflated 42%)
2023-01-11T23:04:19.2543765Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221527.xml (deflated 40%)
2023-01-11T23:04:19.2544543Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221534.xml (deflated 40%)
2023-01-11T23:04:19.2545316Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221541.xml (deflated 41%)
2023-01-11T23:04:19.2546088Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221543.xml (deflated 42%)
2023-01-11T23:04:19.2546848Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221546.xml (deflated 42%)
2023-01-11T23:04:19.2547674Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221550.xml (deflated 41%)
2023-01-11T23:04:19.2548459Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221558.xml (deflated 41%)
2023-01-11T23:04:19.2549294Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221606.xml (deflated 41%)
2023-01-11T23:04:19.2550056Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221614.xml (deflated 42%)
2023-01-11T23:04:19.2550832Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221618.xml (deflated 41%)
2023-01-11T23:04:19.2551603Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221622.xml (deflated 41%)
2023-01-11T23:04:19.2552381Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221629.xml (deflated 40%)
2023-01-11T23:04:19.2553141Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221636.xml (deflated 42%)
2023-01-11T23:04:19.2554039Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221640.xml (deflated 40%)
2023-01-11T23:04:19.2554798Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221647.xml (deflated 41%)
2023-01-11T23:04:19.2555570Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221653.xml (deflated 41%)
2023-01-11T23:04:19.2556348Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221700.xml (deflated 40%)
2023-01-11T23:04:19.2557126Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221707.xml (deflated 43%)
2023-01-11T23:04:19.2557884Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221714.xml (deflated 42%)
2023-01-11T23:04:19.2558665Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221721.xml (deflated 42%)
2023-01-11T23:04:19.2559436Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221728.xml (deflated 42%)
2023-01-11T23:04:19.2560207Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221734.xml (deflated 41%)
2023-01-11T23:04:19.2560979Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221741.xml (deflated 41%)
2023-01-11T23:04:19.2561742Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221748.xml (deflated 42%)
2023-01-11T23:04:19.2562511Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221750.xml (deflated 41%)
2023-01-11T23:04:19.2563288Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221757.xml (deflated 43%)
2023-01-11T23:04:19.2564058Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221800.xml (deflated 44%)
2023-01-11T23:04:19.2565209Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221802.xml (deflated 40%)
2023-01-11T23:04:19.2565991Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221809.xml (deflated 42%)
2023-01-11T23:04:19.2566852Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221811.xml (deflated 42%)
2023-01-11T23:04:19.2567645Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221814.xml (deflated 41%)
2023-01-11T23:04:19.2568407Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221831.xml (deflated 41%)
2023-01-11T23:04:19.2569264Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221838.xml (deflated 41%)
2023-01-11T23:04:19.2570037Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221845.xml (deflated 41%)
2023-01-11T23:04:19.2570806Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221851.xml (deflated 42%)
2023-01-11T23:04:19.2571561Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221854.xml (deflated 41%)
2023-01-11T23:04:19.2572335Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221856.xml (deflated 40%)
2023-01-11T23:04:19.2573110Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221903.xml (deflated 42%)
2023-01-11T23:04:19.2573884Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221910.xml (deflated 41%)
2023-01-11T23:04:19.2574641Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221916.xml (deflated 41%)
2023-01-11T23:04:19.2575415Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221923.xml (deflated 40%)
2023-01-11T23:04:19.2576186Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221930.xml (deflated 40%)
2023-01-11T23:04:19.2576967Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221937.xml (deflated 40%)
2023-01-11T23:04:19.2577727Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221943.xml (deflated 41%)
2023-01-11T23:04:19.2578506Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221950.xml (deflated 41%)
2023-01-11T23:04:19.2579279Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221952.xml (deflated 41%)
2023-01-11T23:04:19.2580051Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221955.xml (deflated 41%)
2023-01-11T23:04:19.2580806Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111221957.xml (deflated 41%)
2023-01-11T23:04:19.2581584Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222004.xml (deflated 41%)
2023-01-11T23:04:19.2582356Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222011.xml (deflated 41%)
2023-01-11T23:04:19.2583134Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222018.xml (deflated 41%)
2023-01-11T23:04:19.2583892Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222024.xml (deflated 41%)
2023-01-11T23:04:19.2584665Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222031.xml (deflated 41%)
2023-01-11T23:04:19.2585435Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222039.xml (deflated 42%)
2023-01-11T23:04:19.2586262Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222047.xml (deflated 41%)
2023-01-11T23:04:19.2587036Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222053.xml (deflated 40%)
2023-01-11T23:04:19.2587873Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222101.xml (deflated 41%)
2023-01-11T23:04:19.2588644Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222109.xml (deflated 41%)
2023-01-11T23:04:19.2589416Z   adding: test/test-reports/dist-gloo/distributed.test_distributed_spawn/TEST-TestDistBackendWithSpawn-20230111222122.xml (deflated 40%)
2023-01-11T23:04:19.2590181Z   adding: test/test-reports/python-unittest/distributed._tools.test_memory_tracker/TEST-TestMemoryTracker-20230111222214.xml (deflated 61%)
2023-01-11T23:04:19.2590970Z   adding: test/test-reports/python-unittest/distributed.elastic.metrics.api_test/TEST-MetricsApiTest-20230111222218.xml (deflated 63%)
2023-01-11T23:04:19.2591751Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.logging_test/TEST-LoggingTest-20230111222222.xml (deflated 55%)
2023-01-11T23:04:19.2592516Z   adding: test/test-reports/python-unittest/distributed.test_launcher/TEST-TestDistributedLaunch-20230111222226.xml (deflated 43%)
2023-01-11T23:04:19.2593267Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_planner/TEST-TestSavePlan-20230111222230.xml (deflated 71%)
2023-01-11T23:04:19.2594071Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_checkpoint_wrapper/TEST-CheckpointWrapperTest-20230111222234.xml (deflated 72%)
2023-01-11T23:04:19.2594979Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.test_megatron_prototype/TEST-TestShardedTensorMegatronLinear-20230111222239.xml (deflated 44%)
2023-01-11T23:04:19.2595876Z   adding: test/test-reports/python-unittest/distributed.elastic.utils.distributed_test/TEST-DistributedUtilTest-20230111222245.xml (deflated 78%)
2023-01-11T23:04:19.2596775Z   adding: test/test-reports/python-unittest/distributed.tensor.parallel.test_view_sharding_dim_change/TEST-TPViewShardingDimChangeTest-20230111222252.xml (deflated 43%)
2023-01-11T23:04:19.2597670Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerServerTest-20230111222259.xml (deflated 71%)
2023-01-11T23:04:19.2598486Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-LocalTimerTest-20230111222259.xml (deflated 69%)
2023-01-11T23:04:19.2599353Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_test/TEST-MultiprocessingRequestQueueTest-20230111222259.xml (deflated 66%)
2023-01-11T23:04:19.2600263Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_embedding_bag/TEST-TestShardedEmbeddingBag-20230111222307.xml (deflated 60%)
2023-01-11T23:04:19.2601102Z   adding: test/test-reports/python-unittest/distributed._shard.sharded_tensor.ops.test_softmax/TEST-TestShardedSoftmax-20230111222316.xml (deflated 59%)
2023-01-11T23:04:19.2601865Z   adding: test/test-reports/python-unittest/distributed._tensor.test_view_ops/TEST-TestViewOps-20230111222325.xml (deflated 51%)
2023-01-11T23:04:19.2602580Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_input/TEST-TestInput-20230111222334.xml (deflated 57%)
2023-01-11T23:04:19.2603358Z   adding: test/test-reports/python-unittest/distributed.elastic.timer.local_timer_example/TEST-LocalTimerExample-20230111222345.xml (deflated 54%)
2023-01-11T23:04:19.2604126Z   adding: test/test-reports/python-unittest/distributed._tensor.test_math_ops/TEST-DistMathOpsTest-20230111222358.xml (deflated 61%)
2023-01-11T23:04:19.2605240Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_apply/TEST-TestApply-20230111222412.xml (deflated 61%)
2023-01-11T23:04:19.2606142Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeOne-20230111222427.xml (deflated 43%)
2023-01-11T23:04:19.2607031Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_overlap/TEST-TestForwardOverlapWorldSizeTwo-20230111222427.xml (deflated 43%)
2023-01-11T23:04:19.2607886Z   adding: test/test-reports/python-unittest/distributed._tensor.test_api/TEST-DTensorAPITest-20230111222442.xml (deflated 75%)
2023-01-11T23:04:19.2608705Z   adding: test/test-reports/python-unittest/distributed.tensor.parallel.test_parallelize_api/TEST-TensorParallelAPITests-20230111222458.xml (deflated 79%)
2023-01-11T23:04:19.2609549Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_hybrid_shard/TEST-TestFSDPHybridShard-20230111222516.xml (deflated 62%)
2023-01-11T23:04:19.2610422Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedReshardOnLoad-20230111222535.xml (deflated 67%)
2023-01-11T23:04:19.2611349Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoad-20230111222535.xml (deflated 46%)
2023-01-11T23:04:19.2612383Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20230111222535.xml (deflated 44%)
2023-01-11T23:04:19.2613350Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111222601.xml (deflated 42%)
2023-01-11T23:04:19.2614196Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111222605.xml (deflated 42%)
2023-01-11T23:04:19.2615030Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111222612.xml (deflated 42%)
2023-01-11T23:04:19.2615862Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111222620.xml (deflated 42%)
2023-01-11T23:04:19.2616694Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111222628.xml (deflated 42%)
2023-01-11T23:04:19.2617531Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_ucc/TEST-TestDistributedNNFunctionsUcc-20230111222636.xml (deflated 41%)
2023-01-11T23:04:19.2618457Z   adding: test/test-reports/python-unittest/distributed.algorithms.ddp_comm_hooks.test_ddp_hooks/TEST-DistributedDataParallelCommHookTest-20230111222643.xml (deflated 79%)
2023-01-11T23:04:19.2619300Z   adding: test/test-reports/python-unittest/distributed._tensor.test_common_rules/TEST-CommonRulesTest-20230111222711.xml (deflated 84%)
2023-01-11T23:04:19.2620074Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_clip_grad_norm/TEST-TestClipGradNorm-20230111222742.xml (deflated 61%)
2023-01-11T23:04:19.2620863Z   adding: test/test-reports/python-unittest/distributed._composable.test_compose/TEST-TestFSDPCheckpoint-20230111222814.xml (deflated 75%)
2023-01-11T23:04:19.2621734Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedReshardOnLoad-20230111222849.xml (deflated 80%)
2023-01-11T23:04:19.2622681Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoad-20230111222849.xml (deflated 54%)
2023-01-11T23:04:19.2623738Z   adding: test/test-reports/python-unittest/distributed.checkpoint.test_file_system_checkpoint_cpu/TEST-TestDistributedStateDictSaveLoadWithSharedTensor-20230111222849.xml (deflated 60%)
2023-01-11T23:04:19.2624634Z   adding: test/test-reports/python-unittest/distributed.algorithms.test_join/TEST-TestJoin-20230111222924.xml (deflated 80%)
2023-01-11T23:04:19.2625483Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111223006.xml (deflated 42%)
2023-01-11T23:04:19.2626383Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111223014.xml (deflated 42%)
2023-01-11T23:04:19.2627248Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111223021.xml (deflated 43%)
2023-01-11T23:04:19.2628169Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111223028.xml (deflated 43%)
2023-01-11T23:04:19.2629019Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111223036.xml (deflated 42%)
2023-01-11T23:04:19.2629863Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111223043.xml (deflated 42%)
2023-01-11T23:04:19.2630696Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111223051.xml (deflated 42%)
2023-01-11T23:04:19.2631537Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111223058.xml (deflated 42%)
2023-01-11T23:04:19.2632385Z   adding: test/test-reports/python-unittest/distributed.test_c10d_spawn_nccl/TEST-TestDistributedNNFunctionsNccl-20230111223105.xml (deflated 42%)
2023-01-11T23:04:19.2633171Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_grad_acc/TEST-TestGradAcc-20230111223112.xml (deflated 93%)
2023-01-11T23:04:19.2633907Z   adding: test/test-reports/python-unittest/distributed._tensor.test_tensor_ops/TEST-DistTensorOpsTest-20230111223211.xml (deflated 84%)
2023-01-11T23:04:19.2634702Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_comm_hooks/TEST-TestCommunicationHooks-20230111223315.xml (deflated 91%)
2023-01-11T23:04:19.2635506Z   adding: test/test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkSubclass-20230111223459.xml (deflated 84%)
2023-01-11T23:04:19.2636288Z   adding: test/test-reports/python-unittest/distributed.test_c10d_pypg/TEST-TestDDPWithWorkWrapper-20230111223459.xml (deflated 84%)
2023-01-11T23:04:19.2637102Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsFQNs-20230111223704.xml (deflated 54%)
2023-01-11T23:04:19.2638050Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsMultipleParamGroups-20230111223704.xml (deflated 83%)
2023-01-11T23:04:19.2638983Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsNoSync-20230111223704.xml (deflated 44%)
2023-01-11T23:04:19.2639882Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsParamAccess-20230111223704.xml (deflated 45%)
2023-01-11T23:04:19.2640803Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsUnshardReshard-20230111223704.xml (deflated 76%)
2023-01-11T23:04:19.2641731Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_use_orig_params/TEST-TestFSDPUseOrigParamsWriteback-20230111223704.xml (deflated 64%)
2023-01-11T23:04:19.2642659Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPDifferentSubmodulePrecision-20230111224002.xml (deflated 77%)
2023-01-11T23:04:19.2643611Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionIgnoredModules-20230111224002.xml (deflated 44%)
2023-01-11T23:04:19.2644927Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionSharded-20230111224002.xml (deflated 92%)
2023-01-11T23:04:19.2645958Z   adding: test/test-reports/python-unittest/distributed.fsdp.test_fsdp_mixed_precision/TEST-TestFSDPMixedPrecisionUnsharded-20230111224002.xml (deflated 64%)
2023-01-11T23:04:19.2646895Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDdpComparisonTest-20230111224457.xml (deflated 41%)
2023-01-11T23:04:19.2647813Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20230111224505.xml (deflated 41%)
2023-01-11T23:04:19.2648800Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20230111224513.xml (deflated 41%)
2023-01-11T23:04:19.2649675Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaDistAutogradTest-20230111224522.xml (deflated 41%)
2023-01-11T23:04:19.2650581Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111224531.xml (deflated 41%)
2023-01-11T23:04:19.2651493Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111224539.xml (deflated 40%)
2023-01-11T23:04:19.2652393Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111224547.xml (deflated 41%)
2023-01-11T23:04:19.2653269Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRemoteModuleTest-20230111224554.xml (deflated 40%)
2023-01-11T23:04:19.2654133Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeCudaRpcTest-20230111224603.xml (deflated 40%)
2023-01-11T23:04:19.2654995Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224612.xml (deflated 40%)
2023-01-11T23:04:19.2655872Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224619.xml (deflated 40%)
2023-01-11T23:04:19.2656750Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224626.xml (deflated 40%)
2023-01-11T23:04:19.2657606Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224634.xml (deflated 40%)
2023-01-11T23:04:19.2658476Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224641.xml (deflated 40%)
2023-01-11T23:04:19.2659339Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224648.xml (deflated 40%)
2023-01-11T23:04:19.2660203Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224655.xml (deflated 40%)
2023-01-11T23:04:19.2661063Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipePipeWithDDPTest-20230111224702.xml (deflated 40%)
2023-01-11T23:04:19.2661985Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224709.xml (deflated 42%)
2023-01-11T23:04:19.2662958Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224720.xml (deflated 42%)
2023-01-11T23:04:19.2663925Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224733.xml (deflated 42%)
2023-01-11T23:04:19.2664874Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224746.xml (deflated 43%)
2023-01-11T23:04:19.2665833Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224758.xml (deflated 43%)
2023-01-11T23:04:19.2666856Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224809.xml (deflated 43%)
2023-01-11T23:04:19.2667829Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224821.xml (deflated 43%)
2023-01-11T23:04:19.2668855Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224833.xml (deflated 43%)
2023-01-11T23:04:19.2669802Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224845.xml (deflated 43%)
2023-01-11T23:04:19.2670764Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224856.xml (deflated 43%)
2023-01-11T23:04:19.2671729Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224904.xml (deflated 43%)
2023-01-11T23:04:19.2672688Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224911.xml (deflated 43%)
2023-01-11T23:04:19.2673630Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224918.xml (deflated 43%)
2023-01-11T23:04:19.2674589Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224925.xml (deflated 43%)
2023-01-11T23:04:19.2675554Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224933.xml (deflated 43%)
2023-01-11T23:04:19.2676517Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224941.xml (deflated 42%)
2023-01-11T23:04:19.2677474Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224952.xml (deflated 43%)
2023-01-11T23:04:19.2678424Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111224956.xml (deflated 43%)
2023-01-11T23:04:19.2679384Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225017.xml (deflated 43%)
2023-01-11T23:04:19.2680345Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225031.xml (deflated 42%)
2023-01-11T23:04:19.2681302Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225043.xml (deflated 43%)
2023-01-11T23:04:19.2682245Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225051.xml (deflated 42%)
2023-01-11T23:04:19.2683207Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225100.xml (deflated 42%)
2023-01-11T23:04:19.2684173Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225109.xml (deflated 42%)
2023-01-11T23:04:19.2685498Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225119.xml (deflated 43%)
2023-01-11T23:04:19.2686444Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225123.xml (deflated 43%)
2023-01-11T23:04:19.2687483Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225133.xml (deflated 42%)
2023-01-11T23:04:19.2688463Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225144.xml (deflated 42%)
2023-01-11T23:04:19.2689504Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225154.xml (deflated 43%)
2023-01-11T23:04:19.2690465Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225205.xml (deflated 42%)
2023-01-11T23:04:19.2691405Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225215.xml (deflated 42%)
2023-01-11T23:04:19.2692367Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225226.xml (deflated 42%)
2023-01-11T23:04:19.2693330Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225236.xml (deflated 42%)
2023-01-11T23:04:19.2694292Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225247.xml (deflated 42%)
2023-01-11T23:04:19.2695230Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225257.xml (deflated 42%)
2023-01-11T23:04:19.2696183Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225308.xml (deflated 42%)
2023-01-11T23:04:19.2697146Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225318.xml (deflated 42%)
2023-01-11T23:04:19.2698108Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225329.xml (deflated 42%)
2023-01-11T23:04:19.2699063Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225339.xml (deflated 42%)
2023-01-11T23:04:19.2700006Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225350.xml (deflated 42%)
2023-01-11T23:04:19.2700961Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225400.xml (deflated 42%)
2023-01-11T23:04:19.2701919Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225411.xml (deflated 42%)
2023-01-11T23:04:19.2702876Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225420.xml (deflated 43%)
2023-01-11T23:04:19.2703819Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225431.xml (deflated 42%)
2023-01-11T23:04:19.2704781Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225440.xml (deflated 42%)
2023-01-11T23:04:19.2705739Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225449.xml (deflated 43%)
2023-01-11T23:04:19.2706701Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225500.xml (deflated 43%)
2023-01-11T23:04:19.2707703Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225510.xml (deflated 43%)
2023-01-11T23:04:19.2708656Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225518.xml (deflated 43%)
2023-01-11T23:04:19.2709679Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225525.xml (deflated 43%)
2023-01-11T23:04:19.2710635Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225532.xml (deflated 42%)
2023-01-11T23:04:19.2711587Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225540.xml (deflated 42%)
2023-01-11T23:04:19.2712527Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225548.xml (deflated 42%)
2023-01-11T23:04:19.2713485Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225557.xml (deflated 42%)
2023-01-11T23:04:19.2714449Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225606.xml (deflated 42%)
2023-01-11T23:04:19.2715411Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225615.xml (deflated 42%)
2023-01-11T23:04:19.2716350Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225623.xml (deflated 42%)
2023-01-11T23:04:19.2717306Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225632.xml (deflated 42%)
2023-01-11T23:04:19.2718265Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225641.xml (deflated 42%)
2023-01-11T23:04:19.2719220Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225651.xml (deflated 42%)
2023-01-11T23:04:19.2720175Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225702.xml (deflated 43%)
2023-01-11T23:04:19.2721117Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225709.xml (deflated 43%)
2023-01-11T23:04:19.2722073Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225720.xml (deflated 43%)
2023-01-11T23:04:19.2723031Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225727.xml (deflated 43%)
2023-01-11T23:04:19.2723987Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225734.xml (deflated 42%)
2023-01-11T23:04:19.2725376Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225742.xml (deflated 42%)
2023-01-11T23:04:19.2726346Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225751.xml (deflated 42%)
2023-01-11T23:04:19.2727308Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225759.xml (deflated 42%)
2023-01-11T23:04:19.2728266Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225806.xml (deflated 42%)
2023-01-11T23:04:19.2729309Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225819.xml (deflated 42%)
2023-01-11T23:04:19.2730269Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225832.xml (deflated 42%)
2023-01-11T23:04:19.2731313Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225846.xml (deflated 41%)
2023-01-11T23:04:19.2732272Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225858.xml (deflated 42%)
2023-01-11T23:04:19.2733227Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225917.xml (deflated 42%)
2023-01-11T23:04:19.2734172Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225937.xml (deflated 42%)
2023-01-11T23:04:19.2735130Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111225942.xml (deflated 42%)
2023-01-11T23:04:19.2736089Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230002.xml (deflated 42%)
2023-01-11T23:04:19.2737048Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230021.xml (deflated 42%)
2023-01-11T23:04:19.2737998Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230039.xml (deflated 42%)
2023-01-11T23:04:19.2738939Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230057.xml (deflated 42%)
2023-01-11T23:04:19.2739897Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230115.xml (deflated 42%)
2023-01-11T23:04:19.2740859Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230134.xml (deflated 42%)
2023-01-11T23:04:19.2741812Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230152.xml (deflated 42%)
2023-01-11T23:04:19.2742749Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230213.xml (deflated 42%)
2023-01-11T23:04:19.2743708Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230231.xml (deflated 42%)
2023-01-11T23:04:19.2744670Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230251.xml (deflated 43%)
2023-01-11T23:04:19.2745629Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeAgentCudaRpcTest-20230111230301.xml (deflated 43%)
2023-01-11T23:04:19.2746610Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20230111230312.xml (deflated 44%)
2023-01-11T23:04:19.2747593Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20230111230319.xml (deflated 44%)
2023-01-11T23:04:19.2748592Z   adding: test/test-reports/python-unittest/distributed.rpc.cuda.test_tensorpipe_agent/TEST-TensorPipeTensorPipeCudaDistAutogradTest-20230111230327.xml (deflated 43%)
2023-01-11T23:04:19.2749414Z   adding: test/test-reports/cpp-distributed/test_distributed/FileStoreTest.xml (deflated 71%)
2023-01-11T23:04:19.2750013Z   adding: test/test-reports/cpp-distributed/test_distributed/HashStoreTest.xml (deflated 71%)
2023-01-11T23:04:19.2750585Z   adding: test/test-reports/cpp-distributed/test_distributed/TCPStoreTest.xml (deflated 80%)
2023-01-11T23:04:19.2751267Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupGlooTest.xml (deflated 81%)
2023-01-11T23:04:19.2751899Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLTest.xml (deflated 79%)
2023-01-11T23:04:19.2752538Z   adding: test/test-reports/cpp-distributed/test_distributed/ProcessGroupNCCLErrorsTest.xml (deflated 67%)
2023-01-11T23:04:19.2753118Z   adding: test/test-reports/cpp-rpc/test_rpc/test_cpp_rpc.xml (deflated 78%)
2023-01-11T23:04:19.2773708Z ##[group]Run # Remove any previous test reports if they exist
2023-01-11T23:04:19.2774084Z # Remove any previous test reports if they exist
2023-01-11T23:04:19.2774408Z rm -f usage-log-*.zip
2023-01-11T23:04:19.2774786Z # this workflow is also run in bazel build test, but we dont generate usage reports for it
2023-01-11T23:04:19.2775179Z # so check to see if the file exists first
2023-01-11T23:04:19.2775476Z if [ -f 'usage_log.txt' ]; then
2023-01-11T23:04:19.2775811Z     zip "usage-log-${FILE_SUFFIX}.zip" 'usage_log.txt'
2023-01-11T23:04:19.2776104Z fi
2023-01-11T23:04:19.2787921Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T23:04:19.2788217Z env:
2023-01-11T23:04:19.2788461Z   GIT_DEFAULT_BRANCH: master
2023-01-11T23:04:19.2788713Z   GPU_FLAG: --gpus all
2023-01-11T23:04:19.2789085Z   DOCKER_CONTAINER_ID: c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56
2023-01-11T23:04:19.2789564Z   FILE_SUFFIX: test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589560299
2023-01-11T23:04:19.2789905Z ##[endgroup]
2023-01-11T23:04:19.3680709Z   adding: usage_log.txt (deflated 95%)
2023-01-11T23:04:19.3728005Z ##[group]Run seemethere/upload-artifact-s3@v5
2023-01-11T23:04:19.3728299Z with:
2023-01-11T23:04:19.3728581Z   s3-prefix: pytorch/pytorch/3896346758/1/artifact

2023-01-11T23:04:19.3728867Z   retention-days: 14
2023-01-11T23:04:19.3729144Z   if-no-files-found: warn
2023-01-11T23:04:19.3729416Z   path: test-jsons-*.zip
2023-01-11T23:04:19.3729655Z   name: artifact
2023-01-11T23:04:19.3729907Z   s3-bucket: gha-artifacts
2023-01-11T23:04:19.3730174Z   region: us-east-1
2023-01-11T23:04:19.3730389Z env:
2023-01-11T23:04:19.3730630Z   GIT_DEFAULT_BRANCH: master
2023-01-11T23:04:19.3730896Z   GPU_FLAG: --gpus all
2023-01-11T23:04:19.3731247Z   DOCKER_CONTAINER_ID: c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56
2023-01-11T23:04:19.3731608Z ##[endgroup]
2023-01-11T23:04:19.8144111Z NOTE: s3-prefix specified, ignoring name parameter
2023-01-11T23:04:19.8144884Z With the provided path, there will be 1 file uploaded
2023-01-11T23:04:19.8145283Z Uploading to s3 prefix: pytorch/pytorch/3896346758/1/artifact
2023-01-11T23:04:19.8157284Z Starting upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589560299.zip
2023-01-11T23:04:19.9667849Z Finished upload of test-jsons-test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589560299.zip
2023-01-11T23:04:19.9823518Z ##[group]Run seemethere/upload-artifact-s3@v5
2023-01-11T23:04:19.9823814Z with:
2023-01-11T23:04:19.9824091Z   s3-prefix: pytorch/pytorch/3896346758/1/artifact

2023-01-11T23:04:19.9824373Z   retention-days: 14
2023-01-11T23:04:19.9824645Z   if-no-files-found: error
2023-01-11T23:04:19.9824925Z   path: test-reports-*.zip
2023-01-11T23:04:19.9825234Z   name: artifact
2023-01-11T23:04:19.9825485Z   s3-bucket: gha-artifacts
2023-01-11T23:04:19.9825745Z   region: us-east-1
2023-01-11T23:04:19.9825958Z env:
2023-01-11T23:04:19.9826194Z   GIT_DEFAULT_BRANCH: master
2023-01-11T23:04:19.9826460Z   GPU_FLAG: --gpus all
2023-01-11T23:04:19.9826811Z   DOCKER_CONTAINER_ID: c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56
2023-01-11T23:04:19.9827291Z ##[endgroup]
2023-01-11T23:04:20.4262377Z NOTE: s3-prefix specified, ignoring name parameter
2023-01-11T23:04:20.4263155Z With the provided path, there will be 1 file uploaded
2023-01-11T23:04:20.4263520Z Uploading to s3 prefix: pytorch/pytorch/3896346758/1/artifact
2023-01-11T23:04:20.4274622Z Starting upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589560299.zip
2023-01-11T23:04:20.6317985Z Finished upload of test-reports-test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589560299.zip
2023-01-11T23:04:20.6473638Z ##[group]Run seemethere/upload-artifact-s3@v5
2023-01-11T23:04:20.6473934Z with:
2023-01-11T23:04:20.6474200Z   s3-prefix: pytorch/pytorch/3896346758/1/artifact

2023-01-11T23:04:20.6474501Z   retention-days: 14
2023-01-11T23:04:20.6474778Z   if-no-files-found: ignore
2023-01-11T23:04:20.6475055Z   path: usage-log-*.zip
2023-01-11T23:04:20.6475290Z   name: artifact
2023-01-11T23:04:20.6475544Z   s3-bucket: gha-artifacts
2023-01-11T23:04:20.6475816Z   region: us-east-1
2023-01-11T23:04:20.6476033Z env:
2023-01-11T23:04:20.6476274Z   GIT_DEFAULT_BRANCH: master
2023-01-11T23:04:20.6476542Z   GPU_FLAG: --gpus all
2023-01-11T23:04:20.6476895Z   DOCKER_CONTAINER_ID: c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56
2023-01-11T23:04:20.6477260Z ##[endgroup]
2023-01-11T23:04:21.0958928Z NOTE: s3-prefix specified, ignoring name parameter
2023-01-11T23:04:21.0959590Z With the provided path, there will be 1 file uploaded
2023-01-11T23:04:21.0959957Z Uploading to s3 prefix: pytorch/pytorch/3896346758/1/artifact
2023-01-11T23:04:21.0971733Z Starting upload of usage-log-test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589560299.zip
2023-01-11T23:04:21.3290958Z Finished upload of usage-log-test-distributed-1-3-linux.8xlarge.nvidia.gpu_10589560299.zip
2023-01-11T23:04:21.3446829Z ##[group]Run # shellcheck disable=SC2156
2023-01-11T23:04:21.3447197Z # shellcheck disable=SC2156
2023-01-11T23:04:21.3447603Z find . -iname "core.[1-9]*" -exec docker exec "${DOCKER_CONTAINER_ID}" sh -c "gdb python {} -ex 'bt' -ex 'q'" \;
2023-01-11T23:04:21.3460767Z shell: /usr/bin/bash -e {0}
2023-01-11T23:04:21.3461026Z env:
2023-01-11T23:04:21.3461255Z   GIT_DEFAULT_BRANCH: master
2023-01-11T23:04:21.3461537Z   GPU_FLAG: --gpus all
2023-01-11T23:04:21.3461912Z   DOCKER_CONTAINER_ID: c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56
2023-01-11T23:04:21.3462253Z ##[endgroup]
2023-01-11T23:04:21.6723543Z ##[group]Run set -x
2023-01-11T23:04:21.6723949Z set -x
2023-01-11T23:04:21.6724560Z python3 -m pip install -r requirements.txt
2023-01-11T23:04:21.6724940Z python3 -m pip install boto3==1.19.12
2023-01-11T23:04:21.6725388Z python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2023-01-11T23:04:21.6738057Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T23:04:21.6738366Z env:
2023-01-11T23:04:21.6738619Z   GIT_DEFAULT_BRANCH: master
2023-01-11T23:04:21.6738876Z   GPU_FLAG: --gpus all
2023-01-11T23:04:21.6739253Z   DOCKER_CONTAINER_ID: c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56
2023-01-11T23:04:21.6739641Z   AWS_DEFAULT_REGION: us-east-1
2023-01-11T23:04:21.6739883Z   BRANCH: 
2023-01-11T23:04:21.6740139Z   TEST_CONFIG: distributed
2023-01-11T23:04:21.6740400Z   SHARD_NUMBER: 1
2023-01-11T23:04:21.6740724Z   BUILD_ENVIRONMENT: linux-bionic-cuda11.7-py3.10-gcc7
2023-01-11T23:04:21.6741064Z   PR_NUMBER: 
2023-01-11T23:04:21.6741307Z   PYTORCH_RETRY_TEST_CASES: 1
2023-01-11T23:04:21.6741599Z   PYTORCH_OVERRIDE_FLAKY_SIGNAL: 1
2023-01-11T23:04:21.6741927Z   SHA1: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T23:04:21.6742213Z   TAG: ciflow/trunk/91627
2023-01-11T23:04:21.6742476Z   WORKFLOW_ID: 3896346758
2023-01-11T23:04:21.6742918Z   GITHUB_TOKEN: ***
2023-01-11T23:04:21.6743176Z   GHA_WORKFLOW_JOB_ID: 10589560299
2023-01-11T23:04:21.6743442Z ##[endgroup]
2023-01-11T23:04:21.6772803Z + python3 -m pip install -r requirements.txt
2023-01-11T23:04:21.9712205Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T23:04:22.0067028Z Requirement already satisfied: astunparse in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 2)) (1.6.3)
2023-01-11T23:04:22.0106110Z Requirement already satisfied: expecttest in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 3)) (0.1.4)
2023-01-11T23:04:22.0117279Z Requirement already satisfied: future in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 4)) (0.18.2)
2023-01-11T23:04:22.0129388Z Requirement already satisfied: hypothesis in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 5)) (6.62.0)
2023-01-11T23:04:22.0660954Z Requirement already satisfied: numpy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 6)) (1.21.6)
2023-01-11T23:04:22.0673067Z Requirement already satisfied: psutil in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 7)) (5.9.1)
2023-01-11T23:04:22.0782609Z Requirement already satisfied: pyyaml in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 8)) (6.0)
2023-01-11T23:04:22.0793634Z Requirement already satisfied: requests in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 9)) (2.26.0)
2023-01-11T23:04:22.1044499Z Requirement already satisfied: setuptools in /usr/lib/python3.7/site-packages (from -r requirements.txt (line 10)) (49.1.3)
2023-01-11T23:04:22.1287570Z Requirement already satisfied: six in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 11)) (1.16.0)
2023-01-11T23:04:22.1299272Z Requirement already satisfied: types-dataclasses in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 12)) (0.6.6)
2023-01-11T23:04:22.1307479Z Requirement already satisfied: typing_extensions in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 13)) (4.4.0)
2023-01-11T23:04:22.1321508Z Requirement already satisfied: sympy in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 14)) (1.10.1)
2023-01-11T23:04:22.1347377Z Requirement already satisfied: filelock in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 15)) (3.9.0)
2023-01-11T23:04:22.1450736Z Requirement already satisfied: networkx in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 16)) (2.6.3)
2023-01-11T23:04:22.1678602Z Requirement already satisfied: jinja2 in /home/ec2-user/.local/lib/python3.7/site-packages (from -r requirements.txt (line 17)) (3.1.2)
2023-01-11T23:04:22.1713662Z Requirement already satisfied: wheel<1.0,>=0.23.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from astunparse->-r requirements.txt (line 2)) (0.38.4)
2023-01-11T23:04:22.1736062Z Requirement already satisfied: attrs>=19.2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (22.2.0)
2023-01-11T23:04:22.2106500Z Requirement already satisfied: sortedcontainers<3.0.0,>=2.1.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (2.4.0)
2023-01-11T23:04:22.2120539Z Requirement already satisfied: exceptiongroup>=1.0.0; python_version < "3.11" in /home/ec2-user/.local/lib/python3.7/site-packages (from hypothesis->-r requirements.txt (line 5)) (1.1.0)
2023-01-11T23:04:22.2144025Z Requirement already satisfied: idna<4,>=2.5; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (3.4)
2023-01-11T23:04:22.2160371Z Requirement already satisfied: charset-normalizer~=2.0.0; python_version >= "3" in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2.0.12)
2023-01-11T23:04:22.2189564Z Requirement already satisfied: urllib3<1.27,>=1.21.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (1.26.14)
2023-01-11T23:04:22.2408982Z Requirement already satisfied: certifi>=2017.4.17 in /home/ec2-user/.local/lib/python3.7/site-packages (from requests->-r requirements.txt (line 9)) (2022.12.7)
2023-01-11T23:04:22.2420769Z Requirement already satisfied: mpmath>=0.19 in /home/ec2-user/.local/lib/python3.7/site-packages (from sympy->-r requirements.txt (line 14)) (1.2.1)
2023-01-11T23:04:22.2500678Z Requirement already satisfied: MarkupSafe>=2.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from jinja2->-r requirements.txt (line 17)) (2.1.1)
2023-01-11T23:04:22.3177483Z + python3 -m pip install boto3==1.19.12
2023-01-11T23:04:22.6169027Z Defaulting to user installation because normal site-packages is not writeable
2023-01-11T23:04:22.6400993Z Requirement already satisfied: boto3==1.19.12 in /home/ec2-user/.local/lib/python3.7/site-packages (1.19.12)
2023-01-11T23:04:22.6472675Z Requirement already satisfied: s3transfer<0.6.0,>=0.5.0 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.5.2)
2023-01-11T23:04:22.6511405Z Requirement already satisfied: jmespath<1.0.0,>=0.7.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (0.10.0)
2023-01-11T23:04:22.6529526Z Requirement already satisfied: botocore<1.23.0,>=1.22.12 in /home/ec2-user/.local/lib/python3.7/site-packages (from boto3==1.19.12) (1.22.12)
2023-01-11T23:04:22.6599376Z Requirement already satisfied: python-dateutil<3.0.0,>=2.1 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (2.8.2)
2023-01-11T23:04:22.6629955Z Requirement already satisfied: urllib3<1.27,>=1.25.4 in /home/ec2-user/.local/lib/python3.7/site-packages (from botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.26.14)
2023-01-11T23:04:22.6856420Z Requirement already satisfied: six>=1.5 in /home/ec2-user/.local/lib/python3.7/site-packages (from python-dateutil<3.0.0,>=2.1->botocore<1.23.0,>=1.22.12->boto3==1.19.12) (1.16.0)
2023-01-11T23:04:22.9390552Z + python3 -m tools.stats.print_test_stats --upload-to-s3 --compare-with-s3 test
2023-01-11T23:07:41.0475244Z [scribe] Scribe access token not provided, sending report via boto3...
2023-01-11T23:07:41.0476136Z ERROR ENCOUNTERED WHEN UPLOADING TO SCRIBE: {"errorMessage":"2023-01-11T23:07:24.693Z d4131ed0-7aa1-48a8-b207-836346a2a8a9 Task timed out after 60.00 seconds"}
2023-01-11T23:07:41.0476490Z 
2023-01-11T23:07:41.0477025Z ----- Historic stats comparison result ------
2023-01-11T23:07:41.0479864Z 
2023-01-11T23:07:41.0480351Z     job: linux-bionic-cuda11.7-py3.10-gcc7
2023-01-11T23:07:41.0480740Z     commit: 8419ddda87c8a47eacc63b54bc7ec98c1f27c26e
2023-01-11T23:07:41.0480953Z 
2023-01-11T23:07:41.0484718Z Commit graph (base is most recent master ancestor with at least one S3 report):
2023-01-11T23:07:41.0485355Z 
2023-01-11T23:07:41.0485488Z     : (master)
2023-01-11T23:07:41.0485724Z     |
2023-01-11T23:07:41.0486013Z     | * 8419ddda87 (HEAD)            total time  3093.16s
2023-01-11T23:07:41.0486269Z     | |
2023-01-11T23:07:41.0486493Z     | : (2 commits)
2023-01-11T23:07:41.0486718Z     |/
2023-01-11T23:07:41.0491727Z     * db2a237763 (base)  11 reports, total time  4966.48s ± 3495.23s
2023-01-11T23:07:41.0492270Z     * 2b0abd4ce3         11 reports, total time  4990.75s ± 3463.36s
2023-01-11T23:07:41.0492777Z     * f7939b21e1         33 reports, total time  3500.97s ± 3537.57s
2023-01-11T23:07:41.0493216Z     * cb3204823e         11 reports, total time  4951.99s ± 3458.09s
2023-01-11T23:07:41.0493619Z     * 6e236553f5         11 reports, total time  4964.39s ± 3513.19s
2023-01-11T23:07:41.0494043Z     * cce577b391         11 reports, total time  4938.35s ± 3358.29s
2023-01-11T23:07:41.0494473Z     * fae821c2f1         11 reports, total time  4751.06s ± 3169.39s
2023-01-11T23:07:41.0494876Z     * 0c3659586d         11 reports, total time  4713.33s ± 3185.77s
2023-01-11T23:07:41.0495294Z     * 122245985a         11 reports, total time  4767.91s ± 3184.40s
2023-01-11T23:07:41.0495712Z     * b797a24259         11 reports, total time  4784.26s ± 3253.10s
2023-01-11T23:07:41.0495967Z     |
2023-01-11T23:07:41.0496178Z     :
2023-01-11T23:07:41.0496313Z 
2023-01-11T23:07:41.0496486Z Removed  (across 1397 suites)     0 tests, totaling     0.00s
2023-01-11T23:07:41.0497060Z Modified (across    0 suites)     0 tests, totaling     0.00s
2023-01-11T23:07:41.0497395Z Added    (across   67 suites)   767 tests, totaling +4111.56s
2023-01-11T23:07:41.1097761Z ##[group]Run pytorch/test-infra/.github/actions/teardown-linux@main
2023-01-11T23:07:41.1098103Z with:
2023-01-11T23:07:41.1098321Z env:
2023-01-11T23:07:41.1098562Z   GIT_DEFAULT_BRANCH: master
2023-01-11T23:07:41.1098815Z   GPU_FLAG: --gpus all
2023-01-11T23:07:41.1099185Z   DOCKER_CONTAINER_ID: c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56
2023-01-11T23:07:41.1099544Z ##[endgroup]
2023-01-11T23:07:41.1116811Z ##[group]Run set -eou pipefail
2023-01-11T23:07:41.1117122Z set -eou pipefail
2023-01-11T23:07:41.1117378Z 
2023-01-11T23:07:41.1117686Z echo "Holding runner for 2 hours until all ssh sessions have logged out"
2023-01-11T23:07:41.1118031Z for _ in $(seq 1440); do
2023-01-11T23:07:41.1118334Z     # Break if no ssh session exists anymore
2023-01-11T23:07:41.1118629Z     if [ "$(who)" = "" ]; then
2023-01-11T23:07:41.1118884Z       break
2023-01-11T23:07:41.1119154Z     fi
2023-01-11T23:07:41.1119376Z     echo "."
2023-01-11T23:07:41.1119622Z     sleep 5
2023-01-11T23:07:41.1119862Z done
2023-01-11T23:07:41.1133329Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T23:07:41.1133636Z env:
2023-01-11T23:07:41.1133880Z   GIT_DEFAULT_BRANCH: master
2023-01-11T23:07:41.1134183Z   GPU_FLAG: --gpus all
2023-01-11T23:07:41.1134558Z   DOCKER_CONTAINER_ID: c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56
2023-01-11T23:07:41.1134913Z ##[endgroup]
2023-01-11T23:07:41.1164490Z Holding runner for 2 hours until all ssh sessions have logged out
2023-01-11T23:07:41.1213370Z ##[group]Run # ignore expansion of "docker ps -q" since it could be empty
2023-01-11T23:07:41.1213800Z # ignore expansion of "docker ps -q" since it could be empty
2023-01-11T23:07:41.1214152Z # shellcheck disable=SC2046
2023-01-11T23:07:41.1214461Z docker stop $(docker ps -q) || true
2023-01-11T23:07:41.1214760Z # Prune all of the docker images
2023-01-11T23:07:41.1215059Z docker system prune -af
2023-01-11T23:07:41.1227074Z shell: /usr/bin/bash --noprofile --norc -e -o pipefail {0}
2023-01-11T23:07:41.1227357Z env:
2023-01-11T23:07:41.1227602Z   GIT_DEFAULT_BRANCH: master
2023-01-11T23:07:41.1227871Z   GPU_FLAG: --gpus all
2023-01-11T23:07:41.1228223Z   DOCKER_CONTAINER_ID: c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56
2023-01-11T23:07:41.1228578Z ##[endgroup]
2023-01-11T23:07:41.7298589Z c3943a31ca1f
2023-01-11T23:07:42.8264484Z Deleted Containers:
2023-01-11T23:07:42.8264916Z c3943a31ca1f211b9a6338b7b0b5feb6cc943ecc4276c46ae74866da43259a56
2023-01-11T23:07:42.8265169Z 
2023-01-11T23:07:47.5817301Z Deleted Images:
2023-01-11T23:07:47.5818477Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7:fd224c2e6c79d7fdec6408da598bf52bc5b201dd
2023-01-11T23:07:47.5819500Z untagged: 308535385114.dkr.ecr.us-east-1.amazonaws.com/pytorch/pytorch-linux-bionic-cuda11.7-cudnn8-py3-gcc7@sha256:0da23f4faf0ce20770149c4a783e08eaa91c07112511dc5511c77937c66edb24
2023-01-11T23:07:47.5820139Z deleted: sha256:dd055998e88c3bb7db98caef99cc4aaaa492114a459a38a5f0ab49c735f40318
2023-01-11T23:07:47.5820590Z deleted: sha256:e4008aa27d9451086197883cfac22b827879bbe380f63c8c39e3db8313773f3c
2023-01-11T23:07:47.5821044Z deleted: sha256:acc638ed73c788f1c8fdbf04e65d27fa42e6c32d67dbdb50616e173ef284a563
2023-01-11T23:07:47.5821493Z deleted: sha256:f5db2d6ac11f27c63a5f2d0250a45efbd078c37a32d2e2973e544ea526501ba9
2023-01-11T23:07:47.5821933Z deleted: sha256:ce58ef265c69d549d5071cb5418ec43a703978b2c7a88b1141673272cd29de77
2023-01-11T23:07:47.5822368Z deleted: sha256:2e1d5c5ea4a63305e9617c6ab380960f330e1755c39c714f3a3eefb2b603e92a
2023-01-11T23:07:47.5822780Z deleted: sha256:f6e5b9727392978f694412774dde4231ab32666b8604ff0d64727308d45a9163
2023-01-11T23:07:47.5823795Z deleted: sha256:0cfd1beb2f29eb03cb53178c729bf68f37f74bbd713aa6e3a6b1dd0b8121eb61
2023-01-11T23:07:47.5824482Z deleted: sha256:8ec9997c9a1e58cb7d553f0862f1f975bdc1bbd0d0297b6c483bf8731508824b
2023-01-11T23:07:47.5824966Z deleted: sha256:367991b9b8c74307719e7037512b4d1d67917a888c40c4ed86af9843f0c38c77
2023-01-11T23:07:47.5825391Z deleted: sha256:83b736f4ed1139df1136ab38e29568f048e4ce111951d4fdc1a508713303ca00
2023-01-11T23:07:47.5825819Z deleted: sha256:d2ca56a8d719ce8cfe4b65b165cdee296b1b456c56eb3e06990ca62ba42bc18a
2023-01-11T23:07:47.5826267Z deleted: sha256:cdfc9a191a57609f700ced389b19973ef4436b66b1d47313c799be166d6fce4b
2023-01-11T23:07:47.5826701Z deleted: sha256:293a82b424c32cb77ae038879dff6856d5fd08e5b9db2c5b015f424ebf88d24b
2023-01-11T23:07:47.5827127Z deleted: sha256:21a63ac2015ba16ccb51b9090e0f4b78a8437ca4dd189f376d1e9f45e6c74d3e
2023-01-11T23:07:47.5827574Z deleted: sha256:afc58208e3d75e15f79b0f474b361bbc1e21b8eb232fe613a1c79b8415827c86
2023-01-11T23:07:47.5828004Z deleted: sha256:86179776c9e36c00cf4c4a64f717303555b00dc594ab664fb29cfdb707eece2b
2023-01-11T23:07:47.5828407Z deleted: sha256:45f06a2900391711a143b22a31156d3013a0f37430ff8367ab4dbb27ac33381e
2023-01-11T23:07:47.5828807Z deleted: sha256:d7d350150f2a9a9ac81a59d44fd01f74d44f5f354f07a2f055aea97c8be52d92
2023-01-11T23:07:47.5829261Z deleted: sha256:eae59ad1f09ea2b1f99a568bfd5f85dae44a3af9b46b5ac5af19931de9e8fb8d
2023-01-11T23:07:47.5829726Z deleted: sha256:d734a02f47eda33d1022c9642ccfc469b44f5b104ab8ae7aa855ea76be550288
2023-01-11T23:07:47.5830167Z deleted: sha256:643cbebfdfe0cee71fed16a3233b2ee4b6392d91833a41269cdc80c8d0841ae9
2023-01-11T23:07:47.5830623Z deleted: sha256:3a4db9e7d414af3a157be5297f5b7bcddc4c63c0e83221d36728ddc32f84bc2d
2023-01-11T23:07:47.5831076Z deleted: sha256:d93fc8ff6f3d356d012ef7cca3d02018fa3c0a4be9ca2ae4ce78d174274ce530
2023-01-11T23:07:47.5831534Z deleted: sha256:cdd9af1450d9ea8f070e7c03dfe076d49ab615a4d3558be68bd7a4f16d804038
2023-01-11T23:07:47.5831961Z deleted: sha256:de611f0f78a22b1c6c68620370fb99a959c668d8c38a2cc832912e784f94869f
2023-01-11T23:07:47.5832381Z deleted: sha256:0c7878e2d089271e8e5c181eef49d0a43c99b827d1a60dafd15a54627d9146b4
2023-01-11T23:07:47.5832815Z deleted: sha256:55fe9bdaa629b37baee75e1f6878ab1941e000e0937ef45452a9482f29de4577
2023-01-11T23:07:47.5833243Z deleted: sha256:1e641da8b29087f06ad852506d00c1eaaedaeed0bf2a451c1d462c0b476c169f
2023-01-11T23:07:47.5833668Z deleted: sha256:77079a0a22a485736fdd6052b5270d4fb8ee1771976ed0d55e0f315cbc6d1da5
2023-01-11T23:07:47.5834087Z deleted: sha256:b3dda998ed6389c88d249f3aa8d96b2f94876931c45284ef00425cdea77b7c07
2023-01-11T23:07:47.5834572Z deleted: sha256:7dcf69834443047244edadb0a7016bc391279d485de12e8d2d8aad25af532912
2023-01-11T23:07:47.5834999Z deleted: sha256:932db4f0d0b27a0b32d81f5e108fdec1c5433e7b4614cfdc39d64189a59bc228
2023-01-11T23:07:47.5835449Z deleted: sha256:99ea7bfa823ad0725ffcc42e5dd47b90270c3a3e49a0b2a31adae4497d029331
2023-01-11T23:07:47.5835896Z deleted: sha256:58c3ab544a412f163fd3613ab991ea85fa1ae7c97f5a6cbc7b86a1b97fdd5484
2023-01-11T23:07:47.5836322Z deleted: sha256:fd75951227e5b4de5b96f5ee360cb3f1caf2f32a33ca89976013a385d23a34be
2023-01-11T23:07:47.5836757Z deleted: sha256:980d2a371b758adf16fc78370ed6b8bcf77846721ef4f20da94a6d1299457ad6
2023-01-11T23:07:47.5837179Z deleted: sha256:311c72e743e14e8490d04f4331dcc0a35309e9b94266986b0b5badd5fe499765
2023-01-11T23:07:47.5837591Z deleted: sha256:857d2698f0a0af2341318f6ed93060f21d498e989b339e2867c5236bab9c63d5
2023-01-11T23:07:47.5837985Z deleted: sha256:df69730c501d9b3ce0f2316b2b638e20515ce1e9aad01098b13234f4a2154927
2023-01-11T23:07:47.5838417Z deleted: sha256:288a6d7efd5c9e470341b16b3ea2cd41124c769de8d10643ac688d651e9767c9
2023-01-11T23:07:47.5838847Z deleted: sha256:6a565f7b04668396c32d11cad845e1cd1b84d09e6f970457c5494adfde59690f
2023-01-11T23:07:47.5839271Z deleted: sha256:b4e45ce76a79fe9f3dbbf723401c4bf189f0e70bc81fc2dbe4e70c80044c2fac
2023-01-11T23:07:47.5839714Z deleted: sha256:e4a3cd7f0f84ce7171b04c994248707950295feb437bc5feaadcb66a3f7bf5a3
2023-01-11T23:07:47.5840239Z deleted: sha256:24ac908f9f592af03d6a51011147428f9e682c79b8ca2ad5afd2b1a44aeed617
2023-01-11T23:07:47.5840741Z deleted: sha256:50cc7186fda7f64aa964824a32c139a1085ce030491c0da5fab99fdecae66fdf
2023-01-11T23:07:47.5841163Z deleted: sha256:6e9c802974cfa887b7300715840ef7aaf5765df415d8d4680f72c6034a10292b
2023-01-11T23:07:47.5841565Z deleted: sha256:30109e8b967541225d66d116256e57334c2b63c25b456f9f7cd72d14d46d8da3
2023-01-11T23:07:47.5841976Z deleted: sha256:18ce8ec73f72efbc789a00688f5d57c798690e22048389f236dbc593cec31d6e
2023-01-11T23:07:47.5842385Z deleted: sha256:195741932e0b070b4fed22eee8d97719dc71f1f569594b418d777b87dbe76a6d
2023-01-11T23:07:47.5842812Z deleted: sha256:6f099faae794c47a468400004f89aed66ec84fa1bd6c606a9877ab09c84a5289
2023-01-11T23:07:47.5843241Z deleted: sha256:5bddaa98761511a0e16047132a49704d0cf176bec84f42b91644b8e7adb3cb88
2023-01-11T23:07:47.5843637Z deleted: sha256:5089072a88c6788d2594696a16346c495f97fd117430602f033541a0f333de5f
2023-01-11T23:07:47.5844027Z deleted: sha256:9bc67bb187c368480f186819831faa7998ba6d4f2e4ab8bd5b5fbc8a5aada045
2023-01-11T23:07:47.5844877Z deleted: sha256:45bbe3d22998589317c7f6c4dd591475423bb37ca9b922529c5878653483b18d
2023-01-11T23:07:47.5845116Z 
2023-01-11T23:07:47.5845298Z Total reclaimed space: 19.53GB
2023-01-11T23:07:47.5899156Z Post job cleanup.
2023-01-11T23:07:47.5936471Z Post job cleanup.
2023-01-11T23:07:47.7299718Z [command]/usr/bin/git version
2023-01-11T23:07:47.7356239Z git version 2.38.1
2023-01-11T23:07:47.7419451Z Temporarily overriding HOME='/home/ec2-user/actions-runner/_work/_temp/5786c9bc-47df-46a5-92e4-6bfb57586b71' before making global git config changes
2023-01-11T23:07:47.7421174Z Adding repository directory to the temporary git global config as a safe directory
2023-01-11T23:07:47.7426399Z [command]/usr/bin/git config --global --add safe.directory /home/ec2-user/actions-runner/_work/pytorch/pytorch
2023-01-11T23:07:47.7467572Z [command]/usr/bin/git config --local --name-only --get-regexp core\.sshCommand
2023-01-11T23:07:47.7505574Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'core\.sshCommand' && git config --local --unset-all 'core.sshCommand' || :
2023-01-11T23:07:47.7831630Z Entering 'android/libs/fbjni'
2023-01-11T23:07:47.7874637Z Entering 'third_party/FP16'
2023-01-11T23:07:47.7919338Z Entering 'third_party/FXdiv'
2023-01-11T23:07:47.7962105Z Entering 'third_party/NNPACK'
2023-01-11T23:07:47.8007481Z Entering 'third_party/QNNPACK'
2023-01-11T23:07:47.8049958Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T23:07:47.8092108Z Entering 'third_party/XNNPACK'
2023-01-11T23:07:47.8146406Z Entering 'third_party/benchmark'
2023-01-11T23:07:47.8188415Z Entering 'third_party/cpuinfo'
2023-01-11T23:07:47.8232736Z Entering 'third_party/cub'
2023-01-11T23:07:47.8276445Z Entering 'third_party/cudnn_frontend'
2023-01-11T23:07:47.8325603Z Entering 'third_party/cutlass'
2023-01-11T23:07:47.8376195Z Entering 'third_party/eigen'
2023-01-11T23:07:47.8420477Z Entering 'third_party/fbgemm'
2023-01-11T23:07:47.8463481Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T23:07:47.8506780Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T23:07:47.8548648Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T23:07:47.8590250Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T23:07:47.8633364Z Entering 'third_party/flatbuffers'
2023-01-11T23:07:47.8678241Z Entering 'third_party/fmt'
2023-01-11T23:07:47.8720990Z Entering 'third_party/foxi'
2023-01-11T23:07:47.8763564Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T23:07:47.8806474Z Entering 'third_party/gloo'
2023-01-11T23:07:47.8848718Z Entering 'third_party/googletest'
2023-01-11T23:07:47.8893187Z Entering 'third_party/ideep'
2023-01-11T23:07:47.8936188Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T23:07:47.8981034Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T23:07:47.9031269Z Entering 'third_party/ios-cmake'
2023-01-11T23:07:47.9074614Z Entering 'third_party/ittapi'
2023-01-11T23:07:47.9117746Z Entering 'third_party/kineto'
2023-01-11T23:07:47.9160616Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T23:07:47.9202792Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T23:07:47.9247139Z Entering 'third_party/nccl/nccl'
2023-01-11T23:07:47.9289396Z Entering 'third_party/neon2sse'
2023-01-11T23:07:47.9331404Z Entering 'third_party/nlohmann'
2023-01-11T23:07:47.9376232Z Entering 'third_party/onnx'
2023-01-11T23:07:47.9434398Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T23:07:47.9477046Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T23:07:47.9521129Z Entering 'third_party/onnx-tensorrt'
2023-01-11T23:07:47.9562622Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T23:07:47.9610034Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T23:07:47.9652807Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T23:07:47.9693851Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T23:07:47.9740521Z Entering 'third_party/pocketfft'
2023-01-11T23:07:47.9784239Z Entering 'third_party/protobuf'
2023-01-11T23:07:47.9829887Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T23:07:47.9871626Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T23:07:47.9915255Z Entering 'third_party/psimd'
2023-01-11T23:07:47.9957957Z Entering 'third_party/pthreadpool'
2023-01-11T23:07:47.9999895Z Entering 'third_party/pybind11'
2023-01-11T23:07:48.0042332Z Entering 'third_party/python-enum'
2023-01-11T23:07:48.0085367Z Entering 'third_party/python-peachpy'
2023-01-11T23:07:48.0128580Z Entering 'third_party/python-six'
2023-01-11T23:07:48.0172712Z Entering 'third_party/sleef'
2023-01-11T23:07:48.0215488Z Entering 'third_party/tbb'
2023-01-11T23:07:48.0259761Z Entering 'third_party/tensorpipe'
2023-01-11T23:07:48.0301951Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T23:07:48.0344129Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T23:07:48.0385162Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T23:07:48.0427429Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T23:07:48.0468415Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T23:07:48.0512890Z Entering 'third_party/zstd'
2023-01-11T23:07:48.0572724Z [command]/usr/bin/git config --local --name-only --get-regexp http\.https\:\/\/github\.com\/\.extraheader
2023-01-11T23:07:48.0601955Z http.https://github.com/.extraheader
2023-01-11T23:07:48.0612281Z [command]/usr/bin/git config --local --unset-all http.https://github.com/.extraheader
2023-01-11T23:07:48.0649028Z [command]/usr/bin/git submodule foreach --recursive git config --local --name-only --get-regexp 'http\.https\:\/\/github\.com\/\.extraheader' && git config --local --unset-all 'http.https://github.com/.extraheader' || :
2023-01-11T23:07:48.0963601Z Entering 'android/libs/fbjni'
2023-01-11T23:07:48.0989323Z http.https://github.com/.extraheader
2023-01-11T23:07:48.1022390Z Entering 'third_party/FP16'
2023-01-11T23:07:48.1047777Z http.https://github.com/.extraheader
2023-01-11T23:07:48.1080349Z Entering 'third_party/FXdiv'
2023-01-11T23:07:48.1105535Z http.https://github.com/.extraheader
2023-01-11T23:07:48.1138982Z Entering 'third_party/NNPACK'
2023-01-11T23:07:48.1162959Z http.https://github.com/.extraheader
2023-01-11T23:07:48.1195659Z Entering 'third_party/QNNPACK'
2023-01-11T23:07:48.1220822Z http.https://github.com/.extraheader
2023-01-11T23:07:48.1254324Z Entering 'third_party/VulkanMemoryAllocator'
2023-01-11T23:07:48.1278677Z http.https://github.com/.extraheader
2023-01-11T23:07:48.1312487Z Entering 'third_party/XNNPACK'
2023-01-11T23:07:48.1337530Z http.https://github.com/.extraheader
2023-01-11T23:07:48.1382173Z Entering 'third_party/benchmark'
2023-01-11T23:07:48.1407060Z http.https://github.com/.extraheader
2023-01-11T23:07:48.1440136Z Entering 'third_party/cpuinfo'
2023-01-11T23:07:48.1466782Z http.https://github.com/.extraheader
2023-01-11T23:07:48.1499634Z Entering 'third_party/cub'
2023-01-11T23:07:48.1523601Z http.https://github.com/.extraheader
2023-01-11T23:07:48.1556810Z Entering 'third_party/cudnn_frontend'
2023-01-11T23:07:48.1583843Z http.https://github.com/.extraheader
2023-01-11T23:07:48.1624444Z Entering 'third_party/cutlass'
2023-01-11T23:07:48.1649897Z http.https://github.com/.extraheader
2023-01-11T23:07:48.1691581Z Entering 'third_party/eigen'
2023-01-11T23:07:48.1716806Z http.https://github.com/.extraheader
2023-01-11T23:07:48.1754598Z Entering 'third_party/fbgemm'
2023-01-11T23:07:48.1779676Z http.https://github.com/.extraheader
2023-01-11T23:07:48.1813179Z Entering 'third_party/fbgemm/third_party/asmjit'
2023-01-11T23:07:48.1837414Z http.https://github.com/.extraheader
2023-01-11T23:07:48.1871568Z Entering 'third_party/fbgemm/third_party/cpuinfo'
2023-01-11T23:07:48.1898355Z http.https://github.com/.extraheader
2023-01-11T23:07:48.1932252Z Entering 'third_party/fbgemm/third_party/googletest'
2023-01-11T23:07:48.1956132Z http.https://github.com/.extraheader
2023-01-11T23:07:48.1991911Z Entering 'third_party/fbgemm/third_party/hipify_torch'
2023-01-11T23:07:48.2016695Z http.https://github.com/.extraheader
2023-01-11T23:07:48.2052217Z Entering 'third_party/flatbuffers'
2023-01-11T23:07:48.2076746Z http.https://github.com/.extraheader
2023-01-11T23:07:48.2112543Z Entering 'third_party/fmt'
2023-01-11T23:07:48.2139043Z http.https://github.com/.extraheader
2023-01-11T23:07:48.2173329Z Entering 'third_party/foxi'
2023-01-11T23:07:48.2197757Z http.https://github.com/.extraheader
2023-01-11T23:07:48.2232212Z Entering 'third_party/gemmlowp/gemmlowp'
2023-01-11T23:07:48.2257474Z http.https://github.com/.extraheader
2023-01-11T23:07:48.2291065Z Entering 'third_party/gloo'
2023-01-11T23:07:48.2316726Z http.https://github.com/.extraheader
2023-01-11T23:07:48.2353811Z Entering 'third_party/googletest'
2023-01-11T23:07:48.2380014Z http.https://github.com/.extraheader
2023-01-11T23:07:48.2417550Z Entering 'third_party/ideep'
2023-01-11T23:07:48.2443152Z http.https://github.com/.extraheader
2023-01-11T23:07:48.2479024Z Entering 'third_party/ideep/mkl-dnn'
2023-01-11T23:07:48.2505890Z http.https://github.com/.extraheader
2023-01-11T23:07:48.2542899Z Entering 'third_party/ideep/mkl-dnn/third_party/oneDNN'
2023-01-11T23:07:48.2568730Z http.https://github.com/.extraheader
2023-01-11T23:07:48.2612597Z Entering 'third_party/ios-cmake'
2023-01-11T23:07:48.2637532Z http.https://github.com/.extraheader
2023-01-11T23:07:48.2672382Z Entering 'third_party/ittapi'
2023-01-11T23:07:48.2699989Z http.https://github.com/.extraheader
2023-01-11T23:07:48.2735657Z Entering 'third_party/kineto'
2023-01-11T23:07:48.2761701Z http.https://github.com/.extraheader
2023-01-11T23:07:48.2798046Z Entering 'third_party/kineto/libkineto/third_party/fmt'
2023-01-11T23:07:48.2825431Z http.https://github.com/.extraheader
2023-01-11T23:07:48.2862281Z Entering 'third_party/kineto/libkineto/third_party/googletest'
2023-01-11T23:07:48.2888931Z http.https://github.com/.extraheader
2023-01-11T23:07:48.2925628Z Entering 'third_party/nccl/nccl'
2023-01-11T23:07:48.2952780Z http.https://github.com/.extraheader
2023-01-11T23:07:48.2988321Z Entering 'third_party/neon2sse'
2023-01-11T23:07:48.3014641Z http.https://github.com/.extraheader
2023-01-11T23:07:48.3049239Z Entering 'third_party/nlohmann'
2023-01-11T23:07:48.3074975Z http.https://github.com/.extraheader
2023-01-11T23:07:48.3111912Z Entering 'third_party/onnx'
2023-01-11T23:07:48.3138754Z http.https://github.com/.extraheader
2023-01-11T23:07:48.3187610Z Entering 'third_party/onnx/third_party/benchmark'
2023-01-11T23:07:48.3213998Z http.https://github.com/.extraheader
2023-01-11T23:07:48.3247894Z Entering 'third_party/onnx/third_party/pybind11'
2023-01-11T23:07:48.3272609Z http.https://github.com/.extraheader
2023-01-11T23:07:48.3309566Z Entering 'third_party/onnx-tensorrt'
2023-01-11T23:07:48.3335294Z http.https://github.com/.extraheader
2023-01-11T23:07:48.3368613Z Entering 'third_party/onnx-tensorrt/third_party/onnx'
2023-01-11T23:07:48.3393506Z http.https://github.com/.extraheader
2023-01-11T23:07:48.3432001Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/benchmark'
2023-01-11T23:07:48.3458238Z http.https://github.com/.extraheader
2023-01-11T23:07:48.3492175Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11'
2023-01-11T23:07:48.3518723Z http.https://github.com/.extraheader
2023-01-11T23:07:48.3554033Z Entering 'third_party/onnx-tensorrt/third_party/onnx/third_party/pybind11/tools/clang'
2023-01-11T23:07:48.3580960Z http.https://github.com/.extraheader
2023-01-11T23:07:48.3620252Z Entering 'third_party/pocketfft'
2023-01-11T23:07:48.3646917Z http.https://github.com/.extraheader
2023-01-11T23:07:48.3679150Z Entering 'third_party/protobuf'
2023-01-11T23:07:48.3704961Z http.https://github.com/.extraheader
2023-01-11T23:07:48.3742456Z Entering 'third_party/protobuf/third_party/benchmark'
2023-01-11T23:07:48.3767970Z http.https://github.com/.extraheader
2023-01-11T23:07:48.3801063Z Entering 'third_party/protobuf/third_party/googletest'
2023-01-11T23:07:48.3826689Z http.https://github.com/.extraheader
2023-01-11T23:07:48.3861729Z Entering 'third_party/psimd'
2023-01-11T23:07:48.3887707Z http.https://github.com/.extraheader
2023-01-11T23:07:48.3919900Z Entering 'third_party/pthreadpool'
2023-01-11T23:07:48.3945831Z http.https://github.com/.extraheader
2023-01-11T23:07:48.3979636Z Entering 'third_party/pybind11'
2023-01-11T23:07:48.4004546Z http.https://github.com/.extraheader
2023-01-11T23:07:48.4037635Z Entering 'third_party/python-enum'
2023-01-11T23:07:48.4063425Z http.https://github.com/.extraheader
2023-01-11T23:07:48.4096707Z Entering 'third_party/python-peachpy'
2023-01-11T23:07:48.4122926Z http.https://github.com/.extraheader
2023-01-11T23:07:48.4156680Z Entering 'third_party/python-six'
2023-01-11T23:07:48.4183750Z http.https://github.com/.extraheader
2023-01-11T23:07:48.4217555Z Entering 'third_party/sleef'
2023-01-11T23:07:48.4244816Z http.https://github.com/.extraheader
2023-01-11T23:07:48.4280080Z Entering 'third_party/tbb'
2023-01-11T23:07:48.4306233Z http.https://github.com/.extraheader
2023-01-11T23:07:48.4342501Z Entering 'third_party/tensorpipe'
2023-01-11T23:07:48.4369272Z http.https://github.com/.extraheader
2023-01-11T23:07:48.4402765Z Entering 'third_party/tensorpipe/third_party/googletest'
2023-01-11T23:07:48.4428987Z http.https://github.com/.extraheader
2023-01-11T23:07:48.4464061Z Entering 'third_party/tensorpipe/third_party/libnop'
2023-01-11T23:07:48.4491180Z http.https://github.com/.extraheader
2023-01-11T23:07:48.4526948Z Entering 'third_party/tensorpipe/third_party/libuv'
2023-01-11T23:07:48.4553272Z http.https://github.com/.extraheader
2023-01-11T23:07:48.4589702Z Entering 'third_party/tensorpipe/third_party/pybind11'
2023-01-11T23:07:48.4616450Z http.https://github.com/.extraheader
2023-01-11T23:07:48.4650358Z Entering 'third_party/tensorpipe/third_party/pybind11/tools/clang'
2023-01-11T23:07:48.4676567Z http.https://github.com/.extraheader
2023-01-11T23:07:48.4714592Z Entering 'third_party/zstd'
2023-01-11T23:07:48.4741789Z http.https://github.com/.extraheader
2023-01-11T23:07:48.5052224Z Cleaning up orphan processes